Validation of a Discharge Summary Term Search Method to Detect Adverse Events
Bibliographic record
Abstract
OBJECTIVE: Adverse events are poor health outcomes caused by medical care. Measuring them is necessary for quality improvements, but current detection methods are inadequate. We performed this study to validate a previously derived method of adverse event detection using term searching in physician-dictated discharge summaries. DESIGN: This was a retrospective, chart review study of a random sample of 245 adult medicine and surgery patients admitted to a multicampus academic medical center in 2002. MEASUREMENTS: The authors used a commercially available search engine to scan discharge summaries for the presence of 104 terms that potentially indicate an adverse event. Summaries with any of these terms were reviewed by a physician to determine the term's context. Screen-positive summaries had a term that was contextually indicative of an adverse event. We used a two-stage chart review as the gold standard to determine the true presence or absence of an adverse event. RESULTS: The average patient age was 62 years (standard deviation 18.6) and 55% were admitted to a medical service. By gold standard criteria, 48 of 245 patients had an adverse event. Term searching classified 27 cases with an adverse event, with 11 true positives; 218 cases were classified as not having an adverse event, with 181 true negatives. The sensitivity, specificity, and positive and negative predictive values were 0.23 (95% confidence interval [CI]=0.11-0.35), 0.92 (95% CI=0.88-0.96), 0.41 (95% CI=0.25-0.59), and 0.83 (95% CI=95% 0.77-0.97), respectively. CONCLUSION: Although the sensitivity of the method is low, its high specificity means that the method could be used to replace expensive manual chart reviews by nurses.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.050 | 0.194 |
| Meta-epidemiology (narrow) | 0.001 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.006 | 0.002 |
| Science and technology studies | 0.001 | 0.001 |
| Scholarly communication | 0.002 | 0.001 |
| Open science | 0.002 | 0.002 |
| Research integrity | 0.001 | 0.001 |
| Insufficient payload (model declined to judge) | 0.001 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".