Automatic Scoring of Cognition Drawings - Improving prediction accuracy through curriculum learning
Bibliographic record
Abstract
Dementia and related disorders are a major public health concern as they are among the leading causes of death, disability and dependency in the global elderly population. There is currently no known cure for the disease, making the identification and management of risk factors one of the few promising approaches to address this emerging crisis. One way to detect early signs of cognitive decline is through cognitive testing (e.g., the Montreal Cognitive Assessment, MoCA), which includes drawing tasks such as cubes or clocks to assess the visuospatial domain of cognitive function. Recently, this type of test has been used in several large-scale survey studies, such as SHARE, the Survey of Health, Ageing and Retirement in Europe (https://share-eric.eu/). While these tests are adapted from clinical cognitive assessments, in this setting they are used to make population-based estimates, such as the prevalence of dementia symptoms, rather than individual diagnoses. This will allow researchers to look for patterns in early cognitive decline and hopefully identify factors that can slow or delay its progression. SHARE includes tests from the Addenbrooke’s Cognitive Examination III (see Wagner & Douhou, 2021, https://share-eric.eu/fileadmin/user_upload/Methodology_Volumes/SHARE_Methodenband_WEB_Wave8_MFRB.pdf), but unlike the clinical setting, scoring of the drawings is not done by trained clinicians, but by the regular face-to-face interviewers during fieldwork, posing a potential risk to data quality. In a proof of concept (Bethmann et al, 2023, https://doi.org/10.48550/arXiv.2312.16887) based on approximately 2,000 cube drawings from the German SHARE sub-study, we re-scored the drawings using multiple raters and a subsequent arbitration round to arrive at a preliminary ‚ground truth’ score. In comparison, interviewer scoring accuracy was only around 75%, with ‚partially correct‘ or ‚incorrect‘ cubes being particularly difficult to score. We then trained several different deep learning models (AlexNet, VGG, ResNet, ConvNeXt) on the ‚ground truth‘ data, which resulted in a significantly better scoring accuracy of around 85% for the best models (ConvNeXt). While these results are promising, they are not yet sufficient for production use. We are therefore pursuing several avenues to improve scoring accuracy and hence data quality: We have collected approximately 55,000 additional recording booklets with cognition drawings from other SHARE countries, which will be scanned and then pre-processed. We expect that training our deep learning models on this training dataset will significantly improve scoring accuracy due to its larger size alone. At the same time, we want to improve the training procedure by using an approach called ‚curriculum learning‘ (cf. Bengio et al., 2009, https://doi.org/10.1145/1553374.1553380), which is a method to gradually increase the complexity of the data samples during the training process, similar to the way humans learn. We want to evaluate whether and to what extent this helps to increase the prediction accuracy of our models.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.003 | 0.012 |
| Meta-epidemiology (narrow) | 0.002 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.003 | 0.002 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.002 | 0.002 |
| Open science | 0.002 | 0.002 |
| Research integrity | 0.001 | 0.002 |
| Insufficient payload (model declined to judge) | 0.006 | 0.007 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".