Leveraging Dual Usability Methods to Evaluate Clinical Decision Support Among Patients With Traumatic Brain Injury: Mixed Methods Study
Bibliographic record
Abstract
Background: Patients with traumatic brain injury are at an increased risk of developing venous thromboembolism. Clinical decision support systems (CDSSs) may improve the use of venous thromboembolism prophylaxis protocols, yet suffer from poor compliance among end users due to a lack of user-centered design. Objective: The objective of this work was to improve the content, design, and workflow integration of a traumatic brain injury-CDSS based on feedback from experts and end users. Methods: The CDSS was evaluated leveraging a dual usability approach. A set of usability experts (n=3) and trauma providers (n=5) performed heuristic evaluations and usability testing by end users. Data was collected through a triangulation of methods and analyzed using qualitative (thematic) and quantitative (descriptive) analyses. Results: Among the 145 total issues identified across both methods, 66 issues were found to be unique. Of the 66, a total of 17 issues were found by heuristic evaluations, 43 by usability testing by end users, and 6 were found across both methods. Thematic analysis was conducted on the 66 unique issues, which were further assigned to themes and subsequent subthemes. We identified 13 unique themes. The 3 most prevalent themes of 66 issues were lack of supporting evidence (n=17, 26%), operational barriers arising from the test environment (n=11, 17%), formatting inconsistencies, and lack of following standards (n=8, 12%). The system's usability scale survey score was 77.5 (SD 16, 95% CI 57.6-97.4), interpreted as an acceptable or good usability range. The mean response score for Single Ease Questions for all tasks was 5.9 (SD 0.53). Conclusions: Combining expert and end user-driven usability evaluation methods identified a more comprehensive list of issues. This can facilitate the optimization of the traumatic brain injury-CDSS, resulting in improved usability and care management.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.079 | 0.094 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.002 | 0.003 |
| Bibliometrics | 0.004 | 0.003 |
| Science and technology studies | 0.002 | 0.002 |
| Scholarly communication | 0.004 | 0.002 |
| Open science | 0.001 | 0.003 |
| Research integrity | 0.001 | 0.001 |
| Insufficient payload (model declined to judge) | 0.002 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".