Usability evaluation of order sets in a computerised provider order entry system
Bibliographic record
Abstract
BACKGROUND: Computerised provider order entry (CPOE) is an important patient safety intervention that has encountered significant barriers to implementation. The usability of a CPOE system plays a significant role in its acceptance. The authors conducted a heuristic evaluation of a CPOE order set system to uncover existing usability issues prior to implementation. METHODS: A heuristic evaluation methodology was used to evaluate the usability of a CPOE test order set system. There are 10 heuristic principles, such as error prevention, to help users identify and recover from errors. Evaluators included a staff physician with extensive clinical experience, and three engineers with expertise in heuristic evaluation methodology. The results of the heuristic evaluation were used to create a user centred design prototype. RESULTS: 92 unique heuristic violations were found for the CPOE test order set system, including 35 identified by the clinician and at least one engineer, and 57 of the 92 violations (62%) found only by the clinician. All evaluators identified at least one violation of each of the 10 usability heuristics in their analysis of the CPOE system. A user centred design prototype was created to demonstrate changes that could improve usability. INTERPRETATION: The CPOE test order set system had many usability heuristic violations. Many violations were found by a clinician with knowledge of the heuristic evaluation process. Implementation of the CPOE system was deferred and a new user centred design prototype was developed for future study. The authors recommend conducting heuristic evaluations early in the process of designing, selecting and implementing CPOE systems.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.038 | 0.003 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.000 |
| Bibliometrics | 0.000 | 0.001 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.001 |
| Insufficient payload (model declined to judge) | 0.001 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".