Machine learning models predicting risk of revision or secondary knee injury after anterior cruciate ligament reconstruction demonstrate variable discriminatory and accuracy performance: a systematic review
Bibliographic record
Abstract
BACKGROUND: To summarize the statistical performance of machine learning in predicting revision, secondary knee injury, or reoperations following anterior cruciate ligament reconstruction (ACLR), and to provide a general overview of the statistical performance of these models. METHODS: Three online databases (PubMed, MEDLINE, EMBASE) were searched from database inception to February 6, 2024, to identify literature on the use of machine learning to predict revision, secondary knee injury (e.g. anterior cruciate ligament (ACL) or meniscus), or reoperation in ACLR. The authors adhered to the PRISMA and R-AMSTAR guidelines as well as the Cochrane Handbook for Systematic Reviews of Interventions. Demographic data and machine learning specifics were recorded. Model performance was recorded using discrimination, area under the curve (AUC), concordance, calibration, and Brier score. Factors deemed predictive for revision, secondary injury or reoperation were also extracted. The MINORS criteria were used for methodological quality assessment. RESULTS: Nine studies comprising 125,427 patients with a mean follow-up of 5.82 (0.08-12.3) years were included in this review. Two of nine (22.2%) studies served as external validation analyses. Five (55.6%) studies reported on mean AUC (strongest model range 0.77-0.997). Four (44.4%) studies reported mean concordance (strongest model range: 0.67-0.713). Two studies reported on Brier score, calibration intercept, and calibration slope, with values ranging from 0.10 to 0.18, 0.0051-0.006, and 0.96-0.97 amongst highest performing models, respectively. Four studies reported calibration error, with all four studies demonstrating significant miscalibration at either two or five-year follow-ups amongst 10 of 14 models assessed. CONCLUSION: Machine learning models designed to predict the risk of revision or secondary knee injury demonstrate variable discriminatory performance when evaluated with AUC or concordance metrics. Furthermore, there is variable calibration, with several models demonstrating evidence of miscalibration at two or five-year marks. The lack of external validation of existing models limits the generalizability of these findings. Future research should focus on validating current models in addition to developing new multimodal neural networks to improve accuracy and reliability.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.021 | 0.109 |
| Meta-epidemiology (narrow) | 0.002 | 0.001 |
| Meta-epidemiology (broad) | 0.011 | 0.016 |
| Bibliometrics | 0.013 | 0.009 |
| Science and technology studies | 0.001 | 0.001 |
| Scholarly communication | 0.003 | 0.003 |
| Open science | 0.003 | 0.001 |
| Research integrity | 0.002 | 0.001 |
| Insufficient payload (model declined to judge) | 0.002 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".