AI-Powered Drug Classification and Indication Mapping for Pharmacoepidemiologic Studies: Prompt Development and Validation
Bibliographic record
Abstract
BACKGROUND: Pharmacoepidemiologic studies, which promote rational drug use and improve health outcomes, often require Anatomical Therapeutic Chemical Classification System (ATC) drug classification within real-world data (RWD) sources. Existing classification tools are expensive, brittle, or have restrictive terms of service, and lack context that may inform classification itself. OBJECTIVE: This study sought to establish large language models (LLMs) as an assisting technology in the drug classification task. This included developing artificial intelligence prompts that reason about drugs using RWD and showing that the resulting accuracy, efficiency, and effectiveness are favorable to alternative methods. METHODS: A prompt was constructed to classify aspirin as either an analgesic or antithrombotic and evaluated within 12,294 anonymized daily dose strings from a polychronic population residing in the United States and Canada. The patients used a smart medication dispenser called "spencer" and consented to the use of their data for research. The LLM prompt requested that the best and next-best second-level ATC code be returned, and grading was performed on a 3-point scale. After success in a pilot sample of 20, an inference sample of 200 was taken without replacement. Finite population inference was carried out on the proportion of outputs receiving 1 of the top 2 grades. As a benchmark, Google's Programmable Search Engine was used to query the drug name plus "ATC code" followed by regex-based extraction of ATC codes. All imperfect results were reviewed. RESULTS: The population consisted of 12,294 daily dose strings from 86.26% (2908/3371) patients residing in Canada and 13.73% (463/3371) residing in the United States. A prompt using the chain-of-thought reasoning was able to distinguish between aspirin's analgesic versus antithrombotic therapeutic uses and performed well in the pilot sample. In the inferential sample, 87.5% (175/200) were graded as perfect, 5% (10/200) had a minor issue, and 7.5% (15/200) had a major issue. The estimate of the proportion of at least mostly correct classification was 92.5% (185/200, 80% CI 90.1%-94.9%). For the search-based algorithm, 82.5% (165/200) were deemed acceptable. The chain-of-thought reasoning was most helpful with supplements (eg, folic acid) when high doses indicated antianemic preparations. The problem formulation of daily dose inputs and multiple ATC outputs was sometimes incompatible with the drug (eg, pregabalin, calcitriol, and methotrexate). CONCLUSIONS: GPT-4o offers cost-effective drug classification from RWD without violating any terms of service. Using a chain-of-thought prompting technique, GPT-4o can reason about drug dosages that affect the class. The wide accessibility of LLMs gives every research team the ability to classify drugs at scale, a key prerequisite of pharmacoepidemiologic research.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.001 | 0.000 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.000 | 0.000 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".