1301 Effect of attribution on external validation of the EULAR/ACR SLE classification criteria
Bibliographic record
Abstract
Background With their new structure of ever positive anti-nuclear antibodies (ANA) as an obligatory entry criterion and weighted specific criteria with a cut-off of ≥ 10, the European League Against Rheumatism/American College of Rheumatology (EULAR/ACR) 2019 classification criteria for systemic lupus erythematosus (SLE) has a sensitivity of 96.1% and a specificity of 93.4% in the validation cohort.1,2 An analysis of the performance of the individual criteria items found that the specificity of joint involvement was 90.9%, but would drop to 57.6% if the attribution rule was not applied.3 The attribution rule states that only those items should be counted towards classification that have no alternative explanation more likely than SLE. The new criteria have been externally validated in a number of studies. From many of the external validation studies, it is not clear whether this attribution rule was followed Methods A literature search was performed for „lupus criteria‘. Titles and abstracts were screened for studies that (i) referred to the EULAR/ACR criteria (even if using different terms) and (ii) indicated sensitivity and/or specificity estimates. The association between criteria specificity and frequency of joint involvement in the non-SLE control group and association between ANA positivity and criteria sensitivity were evaluated. Results Operating characteristics of the SLE classification criteria have been evaluated in 19 studies. The external validation studies reported a sensitivity range of 84.8-97.6% and specificity range of (58.4-97.3%) (table 1). Specificity was evaluated in 14 studies. In 3 of the studies appropriate use of the attribution rule was apparent. One study was excluded for focusing on neuropsychiatric manifestations. For the remaining 10 populations, there was a significant negative correlation between specificity and joint disease in the non-SLE control population. (r=−0.73, p=0.016), as depicted in figure 1 (left panel). Sensitivity estimates are reported in 19 studies, and the percentage of ANA positive SLE patients was reported for 17 of these. There was a positive correlation between ANA positivity and criteria sensitivity (r=0.50, p=0.043). (figure 1, right panel) Conclusions Specificity of the EULAR/ACR criteria is dependent on the correct use of the attribution rule. Higher percentages of patients with joint involvement in the non-SLE control populations is associated with a lower EULAR/ACR criteria specificity. Since joint involvement is particularly vulnerable to not using attribution, this suggests that the lower specificity in some external validation studies in part is due to not fully applying the attribution rule. Sensitivity was high throughout the analyzed studies. It is therefore crucial to differentiate between classification and diagnosis and keep in mind that not fulfilling SLE classification criteria is no valid argument against diagnosing SLE in an individual patient. References Aringer M, Costenbader K, Daikh D, et al. Ann Rheum Dis 2019; 78: 1151-1159. Aringer M, Costenbader K, Daikh D, et al. Arthritis Rheumatol 2019; 71: 1400-1412. Aringer M, Brinks R, Dörner T, et al. Ann Rheum Dis 2021; 80: 775-781
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.466 | 0.730 |
| Meta-epidemiology (narrow) | 0.001 | 0.002 |
| Meta-epidemiology (broad) | 0.003 | 0.008 |
| Bibliometrics | 0.011 | 0.010 |
| Science and technology studies | 0.001 | 0.004 |
| Scholarly communication | 0.005 | 0.005 |
| Open science | 0.004 | 0.005 |
| Research integrity | 0.002 | 0.002 |
| Insufficient payload (model declined to judge) | 0.007 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".