Protocol for validating an algorithm to identify neurocognitive disorders in Canadian Longitudinal Study on Aging participants: an observational study
Bibliographic record
Abstract
INTRODUCTION: In population-based research, disease ascertainment algorithms can be as accurate as, and less costly than, performing supplementary clinical examinations on selected participants to confirm a diagnosis of a neurocognitive disorder (NCD), but they require cohort-specific validation. To optimise the use of the Canadian Longitudinal Study on Aging (CLSA) to understand the epidemiology and burden of NCDs, the CLSA Memory Study will validate an NCD ascertainment algorithm to identify CLSA participants with these disorders using routinely acquired study data. METHODS AND ANALYSIS: Up to 600 CLSA participants with equal numbers of those likely to have no NCD, mild NCD or major NCD based on prior self-reported physician diagnosis of a memory problem or dementia, medication consumption (ie, cholinesterase inhibitors, memantine) and/or self-reported function will be recruited during the follow-up 3 CLSA evaluations (started August 2021). Participants will undergo an assessment by a study clinician who will also review an informant interview and make a preliminary determination of the presence or absence of an NCD. The clinical assessment and available CLSA data will be reviewed by a Central Review Panel who will make a final categorisation of participants as having (1) no NCD, (2) mild NCD or, (3) major NCD (according to fifth version of the Diagnostic and Statistical Manual of Mental Disorders criteria). These will be used as our gold standard diagnosis to determine if the NCD ascertainment algorithm accurately identifies CLSA participants with an NCD. Weighted Kappa statistics will be the primary measure of agreement. Sensitivity, specificity, the C-statistic and the phi coefficient will also be estimated. ETHICS AND DISSEMINATION: Ethics approval has been received from the institutional research ethics boards for each CLSA Data Collection Site (Université de Sherbrooke, Hamilton Integrated Research Ethics Board, Dalhousie University, Nova Scotia Health Research Ethics Board, University of Manitoba, McGill University, McGill University Health Centre Research Institute, Memorial University of Newfoundland, University of Victoria, Élisabeth Bruyère Research Institute of Ottawa, University of British Columbia, Island Health (Formerly the Vancouver Island Health Authority, Simon Fraser University, Calgary Conjoint Health Research Ethics Board).The results of this work will be disseminated to public health professionals, researchers, health professionals, administrators and policy-makers through journal publications, conference presentations, publicly available reports and presentations to stakeholder groups.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.142 | 0.171 |
| Meta-epidemiology (narrow) | 0.004 | 0.004 |
| Meta-epidemiology (broad) | 0.004 | 0.006 |
| Bibliometrics | 0.005 | 0.007 |
| Science and technology studies | 0.007 | 0.003 |
| Scholarly communication | 0.004 | 0.003 |
| Open science | 0.005 | 0.003 |
| Research integrity | 0.005 | 0.006 |
| Insufficient payload (model declined to judge) | 0.061 | 0.013 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".