Speed and Accuracy of a Point of Care Web-Based Knowledge Resource for Clinicians: A Controlled Crossover Trial
Bibliographic record
Abstract
BACKGROUND: Effective knowledge translation at the point of care requires that clinicians quickly find correct answers to clinical questions, and that they have appropriate confidence in their answers. Web-based knowledge resources can facilitate this process. OBJECTIVE: The objective of our study was to evaluate a novel Web-based knowledge resource in comparison with other available Web-based resources, using outcomes of accuracy, time, and confidence. METHODS: We conducted a controlled, crossover trial involving 59 practicing clinicians. Each participant answered questions related to two clinical scenarios. For one scenario, participants used a locally developed Web-based resource, and for the second scenario, they used other self-selected Web-based resources. The local knowledge resource ("AskMayoExpert") was designed to provide very concise evidence-based answers to commonly asked clinical questions. Outcomes included time to a correct response with at least 80% confidence (primary outcome), accuracy, time, and confidence. RESULTS: Answers were more often accurate when using the local resource than when using other Web-based resources, with odds ratio 6.2 (95% CI 2.6-14.5; P<.001) when averaged across scenarios. Time to find an answer was faster, and confidence in that answer was consistently higher, for the local resource (P<.001). Overconfidence was also less frequent with the local resource. In a time-to-event analysis, the chance of responding correctly with at least 80% confidence was 2.5 times greater when using the local resource than with other resources (95% CI 1.6-3.8; P<.001). CONCLUSIONS: Clinicians using a Web-based knowledge resource designed to provide quick, concise answers at the point of care found answers with greater accuracy and confidence than when using other self-selected Web-based resources. Further study to improve the design and implementation of knowledge resources may improve point of care learning.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.012 | 0.024 |
| Meta-epidemiology (narrow) | 0.002 | 0.001 |
| Meta-epidemiology (broad) | 0.005 | 0.003 |
| Bibliometrics | 0.001 | 0.001 |
| Science and technology studies | 0.001 | 0.002 |
| Scholarly communication | 0.002 | 0.003 |
| Open science | 0.002 | 0.001 |
| Research integrity | 0.004 | 0.003 |
| Insufficient payload (model declined to judge) | 0.006 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".