Assessing ChatGPT's capability in understanding and reporting antiretroviral therapy drug–drug interaction effects: Quantitative and qualitative results from the ACCURATE‐DDI study
Notice bibliographique
Résumé
19 Background: Artificial intelligence platforms such as ChatGPT are transforming access to medication information. However, their ability to accurately identify, describe and provide clinically relevant management guidance for antiretroviral (ARV) drug–drug interactions (DDIs) remains unclear. This study evaluates the accuracy of ChatGPT's analysis of ARV-related DDIs compared to established HIV-specific DDI checkers. Materials and methods: Using ChatGPT4o-mini in November 2024, we tested 94 ARV DDI pairs. Pairs included prescription and non-prescription medications, oral and non-oral routes of administration and pharmacokinetic and pharmacodynamic interactions, with an even distribution of non-interacting, potential, and serious interacting combinations. ChatGPT was queried using the following prompt: ‘Is there a drug interaction between Drug A and Drug B? (a) no interaction, (b) potential interaction and (c) serious interaction.’ Using the HIV/HCV Drug Therapy Guide and the University of Liverpool HIV Drug Interactions Checker as references, we calculated accuracy (overall and stratified by DDI severity), sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV). Qualitative analysis was conducted on responses for a subset of 25 pairs tested with the ChatGPT prompt, ‘Can I take Drug A with Drug B?’ Severity, mechanism, clinical effects and management were scored (0 = incorrect, 1 = mixed, 2 = correct), yielding a composite score (maximum = 8). Responses were independently assessed by HIV pharmacist/pharmacologist reviewers; score discrepancies of 2 between two reviewers were resolved by consensus involving a third reviewer. Results: ChatGPT correctly classified 40.4% (38/94) of DDI pairs, with errors primarily due to false negatives (34/56 errors; 60.7%). Among 31 non-interacting pairs, ChatGPT correctly classified 9 (29%) as true negatives; the remaining 22 (71%) were false positives. Among 32 potential DDI pairs, ChatGPT categorized 29 (91%) correctly; 3 (9%) were false negatives. All 31 serious interaction pairs were incorrectly classified (false negatives). Overall sensitivity was 46.0%, specificity 29.0%, PPV 56.9% and NPV 20.9%. In qualitative analysis, the mean composite score across all pairs was 3.9/8. Among non-interacting pairs (mean composite score 2.7/8), ChatGPT incorrectly warned about non-existent DDIs (mean severity 1/2). Mechanism (0.6/2), clinical effects (0.7/2) and management (0.4/2) were often incorrect. Across potential interaction pairs (mean composite score 4.2/8), ChatGPT was better able to categorize the severity (1.6/2), compared to mechanism (0.7/2), clinical effects (0.9/2) and management (1/2). For serious DDI pairs (mean composite score 4.3/8), ChatGPT showed limited accuracy in identifying severity (0.4/2), despite often correctly identifying mechanism (1.4/2) and clinical effects (1.6/2). Serious DDI management was often incorrect (0.9/2). ChatGPT failed to identify ritonavir-mediated DDIs involving non-CYP3A4 pathways, incorrectly claimed that non-CYP3A4 substrates would be boosted when co-administered with cobicistat and failed to identify pharmacodynamic interactions. Conclusion: ChatGPT demonstrated limited accuracy and specificity, often generating responses that combined correct and incorrect information. Notably, ChatGPT demonstrated a tendency to issue unfounded warnings about non-existent or clinically insignificant drug interactions while underreporting the severity of serious DDIs. ChatGPT is not yet a reliable tool for ARV DDI information, but performance may improve with future updates.
Récupéré en direct depuis OpenAlex et désinversé. Les résumés ne sont pas conservés dans cette base de données : les index inversés représentent 8,6 Go des 9,3 Go de texte de la base, et le serveur dispose de 13 Go libres.
Comment cette classification a été obtenuedéplier
Prédiction machine sur la base complète
Imitation des enseignantsNi prévalence calibrée, ni vérité terrain. Validation humaine à venir. Le volet Gemma est une étiquette directe du modèle pour chaque travail de la base, lue sur la notice réduite au titre. Le volet Codex est un classifieur appris des 10 348 étiquettes directes de Codex et calibré sur les taux pondérés de l'échantillon; les champs sans appui suffisant ne portent aucun appel Codex. Le mode candidate est l'union des deux volets; le consensus est leur intersection. Ces sorties portent le statut machine_predicted_unvalidated et ne sont pas des étiquettes humaines.
Scores du classifieur distillé par catégorie (deux têtes)
| Catégorie | Codex | Gemma |
|---|---|---|
| Métarecherche | 0,057 | 0,205 |
| Méta-épidémiologie (sens strict) | 0,000 | 0,000 |
| Méta-épidémiologie (sens large) | 0,001 | 0,001 |
| Bibliométrie | 0,003 | 0,002 |
| Études des sciences et des technologies | 0,001 | 0,003 |
| Communication savante | 0,002 | 0,002 |
| Science ouverte | 0,001 | 0,006 |
| Intégrité de la recherche | 0,001 | 0,001 |
| Charge utile insuffisante (le modèle a refusé de juger) | 0,002 | 0,000 |
Scores machine (provisoires)
Les deux têtes enseignantes du modèle étudiant, lues sur ce travail. Un score ordonne la base pour la relecture; il n'affirme jamais une catégorie, et le statut de validation accompagne chaque rangée tel quel.
Scores de référence d'un modèle non mature (critères de maturité non atteints, 7 itérations). Un score ordonne; il n'affirme jamais une catégorie.
score_only:v0-immature-baseline · tel quel depuis la passe de notation : score_only signifie que le nombre peut ordonner les travaux, et qu'aucune étiquette de catégorie n'en découleClassification
machine, non validéePrédiction automatique; un appel candidat d’une seule source (Gemma direct ou Codex distillé), pas un consensus.
Le détail, modèle par modèle et score par score, se trouve en fin de page sous « Comment cette classification a été obtenue ».