An Assessment Tool for Brachial Plexus Regional Anesthesia Performance: Establishing Construct Validity and Reliability
Bibliographic record
Abstract
BACKGROUND AND OBJECTIVES: Technical proficiency in regional anesthesia is often determined subjectively through in-training evaluations. Objective assessment tools improve these evaluations by providing criteria for measurement. However, any evaluation instrument needs to be valid and reliable before it is adopted into a curriculum. The purpose of this study is to determine the validity and reliability of a devised assessment of residents performing an interscalene brachial plexus block (ISB). METHODS: In this prospective study, 10 junior trainees and 10 senior trainees were videotaped performing an ISB. Junior trainees were defined as in their first year of anesthetic training and had performed less than 10 ISBs independently. Senior trainees had completed at least 1 year of anesthesia training and had performed greater than 10 ISBs independently. Two blinded expert raters independently evaluated the performance of the ISB using a checklist and global rating scale. Construct validity was established if the assessments were able to reliably discriminate between different levels of training. RESULTS: Senior trainees performed an ISB significantly better than junior trainees when assessed using the global rating scale (P < .05) and checklist (P < .001). The overall interrater reliability for the global rating scores was excellent (r = 0.85, P < .05) and was good for the checklist scores (r = 0.74, P < .05). CONCLUSIONS: Both assessment modalities were valid, in that they reliably discriminated between different levels of training. Objective measures of technical skills are feasible, timely, and improve the validity and reliability of competency assessments.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.016 | 0.042 |
| Meta-epidemiology (narrow) | 0.001 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.002 | 0.001 |
| Science and technology studies | 0.000 | 0.001 |
| Scholarly communication | 0.001 | 0.001 |
| Open science | 0.001 | 0.001 |
| Research integrity | 0.001 | 0.001 |
| Insufficient payload (model declined to judge) | 0.001 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".