CoMoVA - A comprehension measurement framework for visualization systems
Bibliographic record
Abstract
Despite the burgeoning interest shown in visualizations by many disciplines, there yet remains the unresolved question concerning comprehension. Is the concept that is being communicated through the visual easily grasped and clearly interpreted? Visual comprehension is that characteristic of any visualization system, which deals with how efficiently and effectively users are able to grasp the underlying concepts through suitable interactions provided for exploring the visually represented information. Comprehension has been considered a very complex subject, which is intangible and subjective in nature. Assessment of comprehension can help to determine the true usefulness of visualization systems to the intended users. A principal contribution of this research is the formulation of an empirical evaluation framework for systematically assessing comprehension support provided by a visualization system to its intended users. To assess comprehension i.e. to measure this seemingly immeasurable factor of visualization systems, we propose a set of criteria based on a detailed analysis of information flow from the raw data to the cognition of information in human mind. Our comprehension criteria are adapted from the pioneering work of two eminent researchers - Donald A. Norman and Aaron Marcus, who have investigated the issues of human perception and cognition, and visual effectiveness respectively. The proposed criteria have been refined with the help of opinions from experts. To gauge and verify the efficacy of these criteria in a practical sense, they were then applied to a bioinformatics visualization study tool and an immersive art visualization environment. Given the vast variety of users and their visualization goals, it may be noted that it is difficult for one to decide on the effectiveness of different visualization tools/techniques in a context independent fashion. We therefore propose an innovative way of evaluating a visualization technique by encapsulating it in a visualization pattern where it is seen as a solution to the visualization problem in a specific context. These visualization patterns guide the tool users/evaluators to compare, understand and select appropriate visualization tools/techniques. Lastly, we propose a novel framework named as CoMoVA (Comprehension Model for Visualization Assessment) that incorporates 'context of use', visualization patterns, visual design principles and important cognitive principles into a coherent whole that can be used to effectively tell us in a more quantifiable manner the benefits of visual representations and interactions provided by a system to the intended audience. Our approach of evaluation of visualization systems is similar to other questionnaire-based approaches such as SUMI (Software Usability Measurement Inventory), where all the questions deal with the measurement of a common trait. We apply this framework to two static software visualization tools in the software visualization domain to demonstrate the practical benefits of using such a framework.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.021 | 0.054 |
| Meta-epidemiology (narrow) | 0.002 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.003 |
| Bibliometrics | 0.010 | 0.005 |
| Science and technology studies | 0.001 | 0.003 |
| Scholarly communication | 0.008 | 0.010 |
| Open science | 0.003 | 0.005 |
| Research integrity | 0.002 | 0.003 |
| Insufficient payload (model declined to judge) | 0.007 | 0.002 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".