Performance of <scp>ChatGPT</scp> in medical examinations: A systematic review and a meta‐analysis
Bibliographic record
Abstract
The use of ChatGPT, an artificial intelligence (AI) language model, has been described in various scientific and medical applications.1 With its human-like conversation capacity and large quantity of data, ChatGPT has the potential to become an important medical education tool.2 ChatGPT's performance in different medical knowledge examinations has been recently studied in various medical disciplines. However, reported rates of correct answers vary dramatically across different examinations and medical fields.3, 4 We aimed to conduct a meta-analysis of studies reporting ChatGPT's performance in medical examinations with multiple-choice questions. PubMed, Scopus and Web of Science websites were searched for relevant articles from the inception of these databases to 2 June 2023, by use of the word “ChatGPT” (no equivalent Mesh term exists). We manually reviewed every article title and abstract. In case an abstract was not available, we accessed the abstract on the journal's web site. We included all peer-reviewed articles, assessing the performance (number of right answers/number of questions) of ChatGPT in multiple-choice questions in the field of medicine. Exclusion criteria were: (i) performance of ChatGPT not in English language; (ii) a study evaluating performance of ChatGPT in a setting other than multiple-choice question (e.g. open questions, frequently asked questions); and (iii) a study not reporting ChatGPT version 3.5. All review stages were conducted independently by two reviewers (RM and GL). Disagreements were resolved by discussion with a third reviewer (YB). Data were extracted from each included study without modifications and a database was constructed containing the study field of medicine (e.g. dermatology, plastic surgery etc.), cohort size (number of questions answered), number of correct questions, ChatGPT performance (number of correct answers/numbers of answered questions) and 95% CI for performance rate. We used MedCalc Statistical Software version 19.2.6 (MedCalc Software bv, Ostend, Belgium) and OpenMeta[Analyst] for the analysis. The process of literature search and article selection is presented in the Supplementary material. Finally, a total of 19 articles were included in the analysis. Two articles (11%) studied Plastic Surgery examinations, two (11%) studied the United Stated Medical Licensing Examinations (USMLE) and two (11%) studied anaesthesia examinations. All other publications studied different medical fields (Table 1). The median number of questions per examination was 242, ranging from 20 in medical physiology examinations to 3705 in anaesthesia examinations (mean 524.5 ± 847.2 standard deviation, Figure 1, Table 1). Overall performance of ChatGPT ranged from 40% in the biomedical admission test to 100% in a diabetes knowledge questionnaire. The mean performance of ChatGPT was 61.1% (95% CI 56.1%–66.0%. As the literature regarding the performance of ChatGPT in medical education is mounting, summarising the current performance of ChatGPT in medical examinations can provide an insight into the present and future of AI in medical education. Our meta-analysis suggests that ChatGPT correctly answered the majority of multiple-choice questions in medical examinations and demonstrated a performance of approximately a passing grade. Medical education, test preparation services, and medical examinations form a large industrial market.5 Currently, it seems that the use of ChatGPT for examination preparation should be prudent, and preparations for examinations using this platform should be done with the proper caution. A correct response rate of more than 95% may allow ChatGPT to become a reliable educational tool,6 and it is unknown whether future versions will reach this maturity level, as the training data set is not specifically developed with a focus on medical education. Our limitations are the inclusion of only ChatGPT version 3.5 studies, the heterogeneity of the included studies with unreported numbers of choices per examination, and the possible overestimation of the tool performance. Future meta-analyses may include future versions of AI chatbots to provide updated understanding of their role in medical education. None. This research received no external funding. None declared. No ethical approval was needed by the institutional review board as this study analyzed only published available public data and no human patients data was sused. The data that support the findings of this study are available on request from the corresponding author. The data are not publicly available due to privacy or ethical restrictions. Data S1. Please note: The publisher is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.003 | 0.029 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.005 | 0.001 |
| Bibliometrics | 0.004 | 0.002 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.001 | 0.000 |
| Research integrity | 0.000 | 0.001 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".