Residents and Medical Students Correctly Answer Clinical Questions More Often with Google and UpToDate than With PubMed or Ovid MEDLINE
Bibliographic record
Abstract
Objective – To determine which search tool (Google, UpToDate, PubMed or Ovid-MEDLINE) produces more accurate answers for residents, medical students, and attending physicians searching on clinical questions in anesthesiology and critical care. Searcher confidence in the answers and speed with which answers were found were also examined. Design – Randomized study without a control group. Setting – Large university medical center. Subjects –Subjects included 15 fourth year medical students (third and fourth year), 35 residents, and 4 attending physicians volunteered and completed the study. One additional attending withdrew halfway through the study. The authors were unsuccessful in recruiting an equal number of subjects from each group. Methods – A set of eight anesthesia and critical care questions was developed, based on their commonality and importance in clinical practice and their answerability. Four search tools were employed: Google, UpToDate, PubMed, and Ovid MEDLINE. In part I, subjects were given a random set of four of the questions to answer with the search tool(s) of their choice, but could use only one search tool per question. In part II, several weeks later, the same subjects were randomly assigned a search tool with which to answer all 8 questions. The authors state that “for data analysis, PubMed was arbitrarily chosen to be the “reference standard.”” Statistical analysis was used to identify significant differences between PubMed and the other search tools. Main Results – Part I: Subjects choosing a search tool were more likely to find a correct answer with Google or UpToDate. There were no statistically significant differences in confidence with answers between any of the search tools and PubMed. Part II: Though subjects were assigned a search tool, some questions were repeated from part I. For repeated questions, Ovid users (compared to PubMed users) were significantly less likely to find the correct answer for repeated questions. Otherwise, there was no statistically significant difference in questions answered correctly. Confidence did not differ. When asked to answer new questions, subjects using Google and UpToDate were significantly more likely to find a correct answer than PubMed users. UpToDate users were more confident. There was no statistical difference in primary outcome (correct answer with high confidence) between Google, Ovid, and PubMed. Pooled data from parts I and II, removing repeated questions: Subjects using Google and UpToDate were more likely to find correct answers. Confidence was highest among UpToDate users. Average search time per question (limited to 5 minutes per question) in ascending order of time spent was: UpToDate, Google, PubMed, and Ovid. Conclusion – While the number of participants is small, the results suggest that the popular search engine Google and the commercially produced secondary online source UpToDate are more useful and efficient for finding answers to questions arising in anesthesiology and critical care practice than tools focused exclusively on indexing the primary literature.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.016 | 0.104 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.002 |
| Bibliometrics | 0.004 | 0.002 |
| Science and technology studies | 0.000 | 0.001 |
| Scholarly communication | 0.001 | 0.003 |
| Open science | 0.000 | 0.002 |
| Research integrity | 0.001 | 0.001 |
| Insufficient payload (model declined to judge) | 0.010 | 0.002 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".