The Challenges of Using ChatGPT for Clinical Decision Support in Orthopaedic Surgery: A Pilot Study
Bibliographic record
Abstract
BACKGROUND: Artificial intelligence (AI) technologies have recently exploded in both accessibility and applicability, including in health care. Although studies have demonstrated its ability to adequately answer simple patient issues or multiple-choice questions, its capacity for deeper complex decision making within health care is relatively untested. In this study, we aimed to delve into AI's ability to integrate multiple clinical data sources and produce a reasonable assessment and plan, specifically in the setting of an orthopaedic surgery consultant. METHODS: Ten common fractures seen by orthopaedic surgeons in the emergency department were chosen. Consult notes from patients sustaining each of these fractures, seen at a level 1 academic trauma center between 2022 and 2023, were stripped of patient data. The history, physical examination, and imaging interpretations were then given to ChatGPT4 in raw and semistructured formats. The AI was asked to determine an assessment and plan as if it were an orthopaedic surgeon. The generated plans were then compared with the actual clinical course of the patient, as determined by our multispecialty trauma conference. RESULTS: When given both raw and semistructured formats of clinical data, ChatGPT4 determined safe and reasonable plans that included the final clinical outcome of the patient scenario. Evaluating large language models is an ongoing field of research without an established quantitative rubric; therefore, our conclusions rely on subjective comparison. CONCLUSION: When given history, physical examination, and imaging interpretations, ChatGPT is able to synthesize complex clinical data into a reasonable and most importantly safe assessment and plan for common fractures seen by orthopaedic surgeons. Evaluating large language models is an ongoing challenge; however, using actual clinical courses as a "benchmark" for comparison presents a possible avenue for further research.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.042 | 0.155 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.001 | 0.001 |
| Science and technology studies | 0.001 | 0.002 |
| Scholarly communication | 0.003 | 0.004 |
| Open science | 0.003 | 0.004 |
| Research integrity | 0.002 | 0.003 |
| Insufficient payload (model declined to judge) | 0.005 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".