Assessing ChatGPT's capability in understanding and reporting antiretroviral therapy drug–drug interaction effects: Quantitative and qualitative results from the ACCURATE‐DDI study
Bibliographic record
Abstract
19 Background: Artificial intelligence platforms such as ChatGPT are transforming access to medication information. However, their ability to accurately identify, describe and provide clinically relevant management guidance for antiretroviral (ARV) drug–drug interactions (DDIs) remains unclear. This study evaluates the accuracy of ChatGPT's analysis of ARV-related DDIs compared to established HIV-specific DDI checkers. Materials and methods: Using ChatGPT4o-mini in November 2024, we tested 94 ARV DDI pairs. Pairs included prescription and non-prescription medications, oral and non-oral routes of administration and pharmacokinetic and pharmacodynamic interactions, with an even distribution of non-interacting, potential, and serious interacting combinations. ChatGPT was queried using the following prompt: ‘Is there a drug interaction between Drug A and Drug B? (a) no interaction, (b) potential interaction and (c) serious interaction.’ Using the HIV/HCV Drug Therapy Guide and the University of Liverpool HIV Drug Interactions Checker as references, we calculated accuracy (overall and stratified by DDI severity), sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV). Qualitative analysis was conducted on responses for a subset of 25 pairs tested with the ChatGPT prompt, ‘Can I take Drug A with Drug B?’ Severity, mechanism, clinical effects and management were scored (0 = incorrect, 1 = mixed, 2 = correct), yielding a composite score (maximum = 8). Responses were independently assessed by HIV pharmacist/pharmacologist reviewers; score discrepancies of 2 between two reviewers were resolved by consensus involving a third reviewer. Results: ChatGPT correctly classified 40.4% (38/94) of DDI pairs, with errors primarily due to false negatives (34/56 errors; 60.7%). Among 31 non-interacting pairs, ChatGPT correctly classified 9 (29%) as true negatives; the remaining 22 (71%) were false positives. Among 32 potential DDI pairs, ChatGPT categorized 29 (91%) correctly; 3 (9%) were false negatives. All 31 serious interaction pairs were incorrectly classified (false negatives). Overall sensitivity was 46.0%, specificity 29.0%, PPV 56.9% and NPV 20.9%. In qualitative analysis, the mean composite score across all pairs was 3.9/8. Among non-interacting pairs (mean composite score 2.7/8), ChatGPT incorrectly warned about non-existent DDIs (mean severity 1/2). Mechanism (0.6/2), clinical effects (0.7/2) and management (0.4/2) were often incorrect. Across potential interaction pairs (mean composite score 4.2/8), ChatGPT was better able to categorize the severity (1.6/2), compared to mechanism (0.7/2), clinical effects (0.9/2) and management (1/2). For serious DDI pairs (mean composite score 4.3/8), ChatGPT showed limited accuracy in identifying severity (0.4/2), despite often correctly identifying mechanism (1.4/2) and clinical effects (1.6/2). Serious DDI management was often incorrect (0.9/2). ChatGPT failed to identify ritonavir-mediated DDIs involving non-CYP3A4 pathways, incorrectly claimed that non-CYP3A4 substrates would be boosted when co-administered with cobicistat and failed to identify pharmacodynamic interactions. Conclusion: ChatGPT demonstrated limited accuracy and specificity, often generating responses that combined correct and incorrect information. Notably, ChatGPT demonstrated a tendency to issue unfounded warnings about non-existent or clinically insignificant drug interactions while underreporting the severity of serious DDIs. ChatGPT is not yet a reliable tool for ARV DDI information, but performance may improve with future updates.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.057 | 0.205 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.003 | 0.002 |
| Science and technology studies | 0.001 | 0.003 |
| Scholarly communication | 0.002 | 0.002 |
| Open science | 0.001 | 0.006 |
| Research integrity | 0.001 | 0.001 |
| Insufficient payload (model declined to judge) | 0.002 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".