Clinician’s ability to identify neck and low back interventions: an inter-rater chance-corrected agreement pilot study
Why this work is in the frame
A frame that forgets how it found something cannot be audited. These are the routes that admitted this work.
Bibliographic record
Abstract
OBJECTIVE: To estimate inter-rater agreement of physical therapists trained in MDT approach and participating in practice-based evidence (PBE) research to identify 72 physical therapy interventions in video demonstrations on a single model and clinical vignettes. PBE is a well designed observational study and demonstrating clinician observational consistency is an important step in conducting PBE research design. METHODS: Two physical therapists volunteered to participate in pilot reliability testing and seven other physical therapists trained in McKenzie Mechanical Diagnosis and Therapy (MDT) methods volunteered for the inter-rater chance-corrected agreement study. All therapists identified interventions presented within 52 videos and 5 written clinical vignettes describing 20 more intervention techniques. Therapists independently identified all interventions. We assessed inter-rater chance-corrected agreement of therapists' ability to identify intervention techniques using Kappa coefficients with associated 95% confidence intervals and indices for bias and prevalence. RESULTS: Of the 147 kappa coefficients estimated, 7% were ⩽0·6, 10% were >0·6 and ⩽0·8, and 83% were >0·8. Agreement was lowest for identifying cognitive behavioral techniques (median kappa = 0·79). The minimum and maximum prevalence and bias indices were 0·33 and 0·85 and 0 and 0·33, respectively suggesting kappa coefficient estimates were strong. Generalized kappa coefficients ranged from 0·73 to 1·00. DISCUSSION: Results provide evidence that substantial to almost perfect inter-rater agreement could be expected when trained therapists identify physical therapy interventions used for patients with spinal impairments from staged videos and vignettes. This may be helpful to reassure clinicians of the quality of the reporting of intervention(s) performed when conducting multivariable analyses in future pragmatic PBE studies. Additional studies are needed to test whether these results can be validated using larger groups of therapists, trained and not trained in MDT methods, as well as examining different methods to examine inter-rater agreement for identifying diverse interventions commonly used for managing patients during routine practice.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.012 | 0.001 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.000 |
| Bibliometrics | 0.000 | 0.000 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.001 |
| Open science | 0.001 | 0.000 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.004 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from it