Building and validating trend-based multiple sclerosis case definitions: a population-based cohort study for Manitoba, Canada
Bibliographic record
Abstract
OBJECTIVE: This study aims to (1) build and validate model-based case definitions for multiple sclerosis (MS) that use trends (ie, trend-based case definitions) and (2) to apply dynamic classification to identify the average number of data years needed for classification (ie, average trend needed). DESIGN: Retrospective cohort study design. PARTICIPANTS: 608 MS cases and 59 620 MS non-cases. SETTING: Data from 1 April 2004 to 31 March 2022 were obtained from the Manitoba Population Research Data Repository. MS case status was ascertained from homecare records and linked to health data. Trend-based case definitions were constructed using multivariate generalised linear mixed models applied to annual numbers of general and specialist physician visits, hospitalisations and MS healthcare contacts or medication dispensations. Dynamic classification, which ascertains cases and non-cases annually, was used to estimate mean classification time. Classification accuracy performance measures, including sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), proportion correctly classified (PCC) and F1-scores, were compared for trend-based case definitions and a deterministic case definition of 3+MS healthcare contacts or medication dispensations. RESULTS: When applied to the full study period, classification accuracy performance measure estimates for all case definitions exceeded 0.90, except sensitivity and PPV for the trend-based dynamic case definition (0.88, 0.64, respectively). PCC was high for all case definitions (0.94-0.99); F1-scores were lower for the trend-based case definitions compared with the deterministic case definition (0.74-0.93 vs 0.96). Dynamic classification identified 5 years as the average trend needed. When applied to the average trend windows, accuracy estimates for trend-based case definitions were lower than the estimates from the full study period (sensitivity: 0.77-0.89; specificity: 0.90-0.97; PPV: 0.54-0.81; NPV: 0.97-0.99; F1-score: 0.64-0.84). Accuracy estimates for the deterministic case definition remained high, except sensitivity (0.42-0.80). F1-score was variable (0.59-0.89). CONCLUSIONS: Trend-based and deterministic case definitions classifications were similar to a population-based clinician assessment reference standard for multiple measures of classification accuracy. However, accuracy estimates for both trend-based and deterministic case definitions varied as the years of data used for classification were reduced. Dynamic classification appears to be a viable option for identifying the average trend needed for trend-based case definitions.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.005 | 0.014 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.003 | 0.005 |
| Science and technology studies | 0.003 | 0.001 |
| Scholarly communication | 0.002 | 0.001 |
| Open science | 0.002 | 0.001 |
| Research integrity | 0.000 | 0.001 |
| Insufficient payload (model declined to judge) | 0.001 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".