Development and validation of a natural language processing system that extracts cognitive test results from clinical notes
Bibliographic record
Abstract
Abstract Background Cognitive test results from electronic health records (EHRs) are key information for assessing the severity and progression of patients with mild cognitive impairment (MCI) and Alzheimer’s’ disease (AD). However, such information is often recorded in unstructured clinical notes rather than in an administrative database. We developed and validated a natural language processing (NLP) system to extract cognitive test results from clinical notes in the Veterans Affair (VA) Healthcare System. Method An NLP system was developed using regular expression‐based rules and Python to extract results for six tests that have been used most frequently in VA: Mini‐Mental State Exam (MMSE), Montreal Cognitive Assessment (MoCA), Saint Louis University Mental Status Examination (SLUMS), Mini‐cog, Boston Naming Test (BNT), and Benton Visual Retention Test (BVRT). The system extracted test results from each note in two steps: (1) searched a test name, or variations and abbreviation of the test name and, if successful, (2) searched the quantitative results (e.g., 12/30 for MMSE, 74th percentile for BNT) and/or descriptive results (e.g., “borderline impairment” for BVRT) within 5 words or one sentence before or after the test name. To balance the system performance and speed, we developed 3‐8 extraction rules per test based on a manual review of 30‐50 notes for each test. We further validated NLP performance on 6 held‐out datasets (50‐200 notes/test). We automatically sampled the development and held‐out notes by searching the test name and its variations/abbreviation to increase positive cases, i.e., notes that contained the above test results. Result The NLP system achieved 0.72‐0.92 predictive positive values (PPV), 0.96‐1.00 recall, and 0.83‐0.95 F1 scores on the validation test sets (Table 1). In addition, it demonstrated a scalable performance (processed 200,000 notes in 7 min), allowing an extraction of millions of notes from ∼1 million patients within hours. Conclusion Rule‐based NLP can extract cognitive test results with adequate performance and scalable capability for clinical notes from patients with MCI or AD within the VA Healthcare System.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.010 | 0.028 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.002 |
| Bibliometrics | 0.002 | 0.001 |
| Science and technology studies | 0.001 | 0.001 |
| Scholarly communication | 0.003 | 0.003 |
| Open science | 0.002 | 0.002 |
| Research integrity | 0.001 | 0.001 |
| Insufficient payload (model declined to judge) | 0.004 | 0.004 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".