MP34-12 UTILITY AND FEASIBILITY OF VIDEO-BASED ASSESSMENT OF RESIDENTS PERFORMING URETEROSCOPY AND LASER LITHOTRIPSY
Bibliographic record
Abstract
You have accessJournal of UrologySurgical Technology & Simulation: Training & Skills Assessment I (MP34)1 Apr 2020MP34-12 UTILITY AND FEASIBILITY OF VIDEO-BASED ASSESSMENT OF RESIDENTS PERFORMING URETEROSCOPY AND LASER LITHOTRIPSY Yuding Wang*, Kelly Dore, Nathan Wong, Jen Hoogenes, and Edward Matsumoto Yuding Wang*Yuding Wang* More articles by this author , Kelly DoreKelly Dore More articles by this author , Nathan WongNathan Wong More articles by this author , Jen HoogenesJen Hoogenes More articles by this author , and Edward MatsumotoEdward Matsumoto More articles by this author View All Author Informationhttps://doi.org/10.1097/JU.0000000000000878.012AboutPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissionsReprints ShareFacebookLinked InTwitterEmail Abstract INTRODUCTION AND OBJECTIVE: Assessment plays a central role in competency-based medical education. The operating room (OR) is a demanding environment with many competing priorities placed on the primary surgeon, one being the assessment of resident performance. Previous studies show that assessment reliability decreases with increased task complexity. The objective was to explore the utility of video-based assessment of resident performance for ureteroscopy and laser lithotripsy. METHODS: Over a 6-month period, 15 ureteroscopy and laser lithotripsy cases performed by urology residents under staff surgeon guidance were captured on video using a 3-camera setup capturing the surgeon’s view, the endoscopic view, and a 360-degree view of the OR. Following each case, the staff surgeon assessed resident performance using the 11-item entrustment-based Ottawa Surgical Competency Operating Room Score Evaluation (O-SCORE), and evaluated his or her own cognitive load during the case using the Surgical Task Load Index (SURG-TLX). Two independent staff urologists reviewed each video using the same assessment tools. Raters’ results were compared those of the case surgeons. RESULTS: Over 15 cases, 630 minutes of video was collected of 7 residents and 5 staff surgeons. Six cases were completed by junior residents and 9 by senior residents. The interrater reliability between video assessors was highly correlated (k=0.82). Comparisons between intraoperative assessment, video-based assessment, and mean O-SCOREs were also highly correlated (r=0.7). Evaluation of O-SCORE domains for intraoperative performance showed a slight decrease in correlation between the intraoperative assessment and video-based assessment (r=0.68). All assessors (video and staff surgeons) expressed low cognitive load, with mean SURG-TLX scores of 5 (out of a possible 100) (SD ±1.8). Video-based assessors frequently rewound the video during critical steps of each case to evaluate resident performance. CONCLUSIONS: We showed that video-based assessment of resident performance during ureteroscopy and laser lithotripsy using the O-SCORE is useful and feasible, with high interrater reliability among the case surgeon and independent video reviewers. Use of the SURG-TLX provided new insight into the cognitive load of both case surgeons and video assessors. Additional research with other types of surgical cases is required to further explore the use of video-based assessment in the OR. Source of Funding: McMaster Surgical Association FundOntario Clinical Investigator Fund © 2020 by American Urological Association Education and Research, Inc.FiguresReferencesRelatedDetails Volume 203Issue Supplement 4April 2020Page: e507-e508 Advertisement Copyright & Permissions© 2020 by American Urological Association Education and Research, Inc.MetricsAuthor Information Yuding Wang* More articles by this author Kelly Dore More articles by this author Nathan Wong More articles by this author Jen Hoogenes More articles by this author Edward Matsumoto More articles by this author Expand All Advertisement PDF downloadLoading ...
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.004 | 0.018 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.002 | 0.000 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.001 |
| Open science | 0.000 | 0.001 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.007 | 0.002 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".