The evolution of <scp>GRADE</scp> (part 2): Still searching for a theoretical and/or empirical basis for the <scp>GRADE</scp> framework
Bibliographic record
Abstract
RATIONALE, AIMS, AND OBJECTIVES: The GRADE framework has been widely adopted as the preferred method for developing clinical practice recommendations. In the first article of our three part series examining the evolution of GRADE, we showed an absence (in the first two versions of GRADE) of a theoretical basis and/or empirical data to support why the presented criteria for determining the quality of evidence regarding the effect estimate and the components under consideration for determining the strength of the recommendation were included and other criteria/components excluded. Furthermore, often, it was not clear how to operationalize the included criteria/components (and integrate them) when using the framework. In part 2 of this series, we examine if version 3 of GRADE offered improvements on previous versions with respect to a justification scheme and how to operationalize the framework's criteria/components. METHODS: Narrative review. RESULTS: Our examination suggests that version 3 has done little to improve on the justification scheme that sustains GRADE. Still absent is a justification (theoretical and/or empirical) for why the criteria/components were chosen. Likewise, version 3 is still lacking clarity regarding how to implement and integrate the criteria/considerations in the framework (ie, operationalize the framework) when determining the quality of evidence or strength of recommendation. Transparency is now emphasized as the merit of GRADE. However, we are offered no theoretical justification for how the use of GRADE should achieve transparency or empirical evidence to support that transparency is achieved. CONCLUSIONS: While version 3 reveals acknowledgement by the authors of GRADE that the framework is a work in progress, it still lacks a justification scheme (theoretical and/or empirical) to sustain it and clarity in its criteria/components to operationalize it. As was suggested in part 1, such issues limit one's ability to scientifically assess the appropriateness of GRADE for its stated purpose.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.531 | 0.830 |
| Meta-epidemiology (narrow) | 0.002 | 0.004 |
| Meta-epidemiology (broad) | 0.007 | 0.009 |
| Bibliometrics | 0.018 | 0.018 |
| Science and technology studies | 0.005 | 0.020 |
| Scholarly communication | 0.034 | 0.022 |
| Open science | 0.014 | 0.016 |
| Research integrity | 0.017 | 0.034 |
| Insufficient payload (model declined to judge) | 0.005 | 0.003 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; the direct Gemma label and the distilled Codex classifier agree on what is shown here.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".