Measuring the usability and usefulness of online patient decision aids: a demonstration of methods
Bibliographic record
Abstract
Background: As the Internet becomes more important for providing health care information to consumers, decision aid developers are increasingly producing or adapting their tools for the Web. To date, however, there has been little systematic effort to discover how best to make use of this medium when providing patient decision support. The computer usability literature distinguishes between usability (easy to use, find, navigate, etc.), and usefulness (the right information for a specific decision maker) of online information. We will demonstrate a number of methods and techniques for studying the usability and usefulness of online patient decision support, in the context of decision support tools developed for patients with musculoskeletal disorders. Methods: The multimedia presentation will demonstrate a number of qualitative and quantitative techniques, drawn from the computer usability and naturalistic decision making traditions, being used in the Ottawa Patient Decision Support Laboratory. The problems and benefits associated with conducting web-based surveys of decision support users will be discussed, as will the role of expert user evaluations. We will demonstrate how the cognitive walkthrough, a standard usability inspection method used to identify components of a task, can be extended to develop a coding scheme (goals, subgoals, and actions required) against which the performance of individual users can be compared. We will also demonstrate how a portable usability laboratory (or lower-tech, less costly versions thereof) can allow access to a rich variety of data sources, including user session transcripts, experimenter field notes, video of the user, and video screen captures of the user session. Coding this rich variety of information at different levels of fidelity will be discussed and demonstrated. Conclusions: There has been little work done on how patient decision support can most effectively be presented on the Web. We will demonstrate a variety of empirical methods designed to enable decision support researchers to address this gap in the literature.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.069 | 0.134 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.002 | 0.002 |
| Science and technology studies | 0.001 | 0.001 |
| Scholarly communication | 0.002 | 0.002 |
| Open science | 0.002 | 0.004 |
| Research integrity | 0.002 | 0.001 |
| Insufficient payload (model declined to judge) | 0.005 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".