51. Structured assessment format for evaluating operative reports (SAFE-OR) in general surgery
Bibliographic record
Abstract
This study determined the construct validity, inter-rater reliability and internal consistency of a “Structured Assessment Format for Evaluating Operative Reports” (SAFE-OR) in general surgery. The assessment instrument was developed using consensus criteria set forth by the Canadian Association of General Surgeons. It includes a structured assessment and a global quality rating scale. Residents divided into novice and experienced groups viewed and dictated a video-taped laparoscopic sigmoid colectomy. Transcriptions were then graded by blinded, independent faculty evaluators using SAFE-OR. Twenty-one residents participated in the study. Mean structured assessment scores (out of 44) were significantly lower for novice versus experienced residents (23.3 ± 5.2 vs 34.1 ± 6.0, t=0.001). Mean global quality scores (out of 45) were similarly lower for novice residents (25.6 ± 4.7 vs 35.9 ± 7.6, t=0.006). Inter-class correlation coefficients were 0.98 (95% CI 0.96-0.99) for structured assessment and 0.93 (95% CI 0.83-0.97) for global quality scales. Cronbach’s alpha coefficients for internal consistency were 0.85 for structured assessment and 0.96 for global quality assessment scales. SAFE-OR demonstrates significant construct validity, excellent inter-rater reliability and high internal consistency. This tool will allow educators to objectively evaluate the quality of trainee operative reports and ultimately provide a mechanism for implementing, monitoring, and refining curriculum for operative dictation communication skills. Moore R. The dictated operative note: important but is it being taught? Journal of the American College of Surgeons 2000; 190(5):639-40. Novitsky Y, Sing R, et al. Prospective, blinded evaluation of accuracy of operative reports dictated by surgical residents. The American Surgeon 2005; 71(8):627-31. Wanzel K, Ward R, et al. Teaching the surgical craft: From selection to certification. Current Problems in Surgery 2002; 39(6):573-659.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.046 | 0.099 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.002 | 0.001 |
| Science and technology studies | 0.000 | 0.001 |
| Scholarly communication | 0.001 | 0.001 |
| Open science | 0.001 | 0.002 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.003 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".