Late Breaking Abstract - Evaluating the accuracy of state-of-the-art large language models in answering asthma multiple choice and objective structured clinical examination questions
Bibliographic record
Abstract
Background: Large Language Models (LLMs) demonstrate promising clinical applications, but their asthma knowledge has not been thoroughly explored. Additionally, state-of-the-art models newly released in 2025 have yet to be evaluated for asthma clinical knowledge. Objectives: To assess LLM performance in asthma Multiple-Choice Questions (MCQ) and Objective Structure Clinical Examinations (OSCE), and to compare accuracy across a range of state-of-the-art LLMs. Methods: Thirteen LLMs (Asthma GPT, ChatGPT 4o, ChatGPT o1, ChatGPT o3-mini, ChatGPT o3-mini-high, Claude 3.7 Sonnet, DeepSeek V3, Gemini 2.0 Flash, Grok 3, Le Chat, Llama 3, NICE AsthmaBot) were tested (5 iterations each) on 114 asthma MCQs and 3 OSCEs. We compared accuracy between LLMs, and then assessed differences between MCQ vs. OSCE, generic vs. medicine-specific LLMs, open-source vs. proprietary LLMs, and between patient vs. clinician-oriented questions. Results: MCQ accuracy was excellent, with 12/13 models exceeding 85% and 4/13 surpassing 95%. OSCE accuracy was varied, though most models still scored >70%. LLMs performed better in MCQs than OSCEs (91% vs 74%, p<0.001), and ChatGPT o1 ranked highest in both categories. No significant differences were found between ChatGPT variants, between open-source vs. proprietary models, or between generic vs. medicine-specific LLMs. Accuracy was 12.2% higher in patient- vs. clinician-oriented MCQs (p < 0.01). Conclusions: Modern LLMs demonstrate high accuracy across varied asthma-related questions. ChatGPT models were the most consistent. These models may serve as important resources for both patients and proviers in asthma education.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.012 | 0.033 |
| Meta-epidemiology (narrow) | 0.003 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.003 |
| Bibliometrics | 0.002 | 0.001 |
| Science and technology studies | 0.001 | 0.001 |
| Scholarly communication | 0.003 | 0.003 |
| Open science | 0.003 | 0.002 |
| Research integrity | 0.003 | 0.003 |
| Insufficient payload (model declined to judge) | 0.010 | 0.007 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".