Development and Validation of Natural Language Processing (NLP)‐Based Risk Prediction Model for Cognitive Impairment in Geriatric Patients
Bibliographic record
Abstract
Abstract Background Dementia poses a significant global crisis, yet 60% of cases go undetected, particularly among specific sub‐populations. Timely diagnosis is crucial for implementing early intervention strategies. Challenges of current screening tools (e.g., Montreal Cognitive Assessment (MOCA) and Mini‐mental Status Exam (MMSE)) include their rule‐based nature, time‐consuming administration and potential bias against individuals with lower education or different socioeconomic backgrounds. Method In this study, we created a patient cohort within Mount Sinai Health System, identifying individuals at high‐risk for cognitive impairment using MMSE scores 23 and below. Extracting clinical notes from inpatient and outpatient settings, we applied specific sampling logic to select the last five notes for each patient before their MMSE. Utilizing word‐embedding techniques, we vectorized sentences and input these vectors into a deep neural network for Named Entity Recognition to extract clinical concepts (e.g diagnosis, signs and symptoms, medications). A psychiatrist reviewed and selected 337 features based on their frequency in the cohort and clinical relevance. Using these features, we created a map to generate feature vectors, and following a train‐test split, 70% of the data was used to train a classifier using the random forest algorithm. The remaining 30% of the cohort was employed for validation purposes. Result Out of 4,366 patients, with documented MMSE scores from 2011 to 2023, 73% had one MMSE, and 29% scored below 24. The developed model demonstrated an AUC ROC of 0.93. At a threshold of 0.5, it showed a sensitivity of 0.86 and specificity of 0.85 in the validation set. On the test set, the AUC ROC was 0.68, with a sensitivity of 0.62 and specificity of 0.63 at a threshold of 0.5. Conclusion The development and validation of this digital tool exemplify translational research that leverages Real‐World Data (RWD) from EMR. The tool aims to assist clinicians in efficient dementia detection across different stages of the disease. Our next phase involves optimizing the feature engineering, fine‐tune the model, and deploying it in the production environment. This will facilitate automated screening of geriatric patients during clinical encounters at Mount Sinai Hospital, allowing us to measure the tool’s prospective performance.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.002 | 0.004 |
| Meta-epidemiology (narrow) | 0.001 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.001 | 0.000 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.001 | 0.001 |
| Open science | 0.001 | 0.001 |
| Research integrity | 0.001 | 0.001 |
| Insufficient payload (model declined to judge) | 0.002 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".