Developing a standardized tool for interpretation of radiology diagnostic accuracy trials
Bibliographic record
Abstract
Summary \nWithin the health sciences, action research is a methodology well suited to the goal of collaboratively improving practice. As the Royal College of Radiology recommends the use of published clinical trials as guides for achieving higher standards of accuracy, it is important for radiologists to reflect deeply on the results from diagnostic accuracy studies. When the results of the gold standard (or reference standard) are used to confirm a particular diagnosis or disease by comparing the diagnostic accuracy to a newer or index test, this is referred to as diagnostic accuracy research. In the reporting of all research, every effort must be made to reduce the incidence of bias. In 2003, the STARD (Standards for Reporting Diagnostic Accuracy) tool was developed for clinicians to enhance the quality of reporting diagnostic accuracy studies. Based on previous studies, experiential knowledge, and an extensive review of the literature, this research demonstrates that the STARD tool is not being fully optimized. The overall aim of this research was to conduct a work-based project within the department of radiology to develop a revised tool, based on the current STARD, which could then be used to more accurately report and interpret the results of radiology diagnostic accuracy trials. This study was conducted in accordance with participatory action research. \n \nMethods \nThe development of this new reporting tool was conducted in collaboration with a group of physicians, and in two distinct phases. First, a needs assessment was sent to eight radiological experts who had agreed to participate in the study. Based on their responses, and feedback from my mentor and colleagues, the next phase of tool development was done using the Delphi technique, after two rounds of which consensus was met. Each phase and cycle iteration to complete the needs assessment and Delphi technique are synonymous with the cycles of action research. The new reporting tool was named the RadSTARD (Radiology Standards for the Reporting of Diagnostic Accuracy Studies), and an elaboration document was written to provide guidance to the end-user. Radiology residents and Fellows at The Ottawa Hospital were then asked to rate their level of confidence in interpreting a diagnostic accuracy article specific to radiology while referring to the RadSTARD. They were also provided a second diagnostic article, the STARD tool, and an elaboration document for comparison. Data was collected using questionnaires that allowed for additional comments. \n \nFindings \nThe validation phase of the RadSTARD tool was completed via triangulation of data, as both a quantitative and qualitative analysis was completed. The results found no significant statistical difference between the two groups as per the Mann-Whitney and chi-square analysis. Likewise, both physician groups indicated that they found RadSTARD increased their level of confidence when interpreting the diagnostic accuracy article. Concomitantly, when combined, 96% of the two physician groups indicated they would use the tool again. \n \nInterpretation \nThese results may be interpreted as generalizable, as there was no discrepancy or statistical difference found in the results between the radiology residents’ and Fellows’ scores, despite the differences in their level of training. Both groups found the RadSTARD tool and elaboration document to be beneficial to them when interpreting the literature. RadSTARD is thus a reliable tool that can be used to validate the results of diagnostic accuracy studies specific to radiology. It will aid radiologists in reporting and interpreting radiology diagnostic accuracy studies, impacting their practice for generations to come.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.014 | 0.026 |
| Meta-epidemiology (narrow) | 0.000 | 0.001 |
| Meta-epidemiology (broad) | 0.002 | 0.001 |
| Bibliometrics | 0.002 | 0.001 |
| Science and technology studies | 0.002 | 0.002 |
| Scholarly communication | 0.000 | 0.001 |
| Open science | 0.003 | 0.000 |
| Research integrity | 0.001 | 0.001 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".