MP35-19 UTILITY AND FEASIBILITY OF VIDEO-BASED ASSESSMENT FOR URETEROSCOPY AND LASER LITHOTRIPSY
Bibliographic record
Abstract
You have accessJournal of UrologySurgical Technology & Simulation: Training & Skills Assessment II (MP35)1 Apr 2019MP35-19 UTILITY AND FEASIBILITY OF VIDEO-BASED ASSESSMENT FOR URETEROSCOPY AND LASER LITHOTRIPSY Yuding Wang*, Kelly Dore, Dana Russell, Jen Hoogenes, Bobby Shayegan, Nathan Wong, and Edward Matsumoto Yuding Wang*Yuding Wang* More articles by this author , Kelly DoreKelly Dore More articles by this author , Dana RussellDana Russell More articles by this author , Jen HoogenesJen Hoogenes More articles by this author , Bobby ShayeganBobby Shayegan More articles by this author , Nathan WongNathan Wong More articles by this author , and Edward MatsumotoEdward Matsumoto More articles by this author View All Author Informationhttps://doi.org/10.1097/01.JU.0000556009.17548.baAboutPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissionsReprints ShareFacebookLinked InTwitterEmail Abstract INTRODUCTION AND OBJECTIVES: Assessment plays a central role in competency-based medical education. The operating room (OR) is a demanding environment with many competing priorities placed on the primary surgeon, one being the assessment of resident performance. Previous studies show that assessment reliability decreases with increased task complexity. The objective of this study was to explore the utility of video-based assessment of resident performance for ureteroscopy and laser lithotripsy. METHODS: Over a 6-month period, 15 ureteroscopy and laser lithotripsy cases performed by urology residents under staff surgeon guidance were captured on video using a 3-camera setup capturing the surgeon’s view, the endoscopic view, and a 360-degree view of the OR. Following the case, the staff surgeon assessed resident performance using the 11-item entrustment-based Ottawa Surgical Competency Operating Room Score Evaluation (O-SCORE), and evaluated his or her own cognitive load during the case using the Surgical Task Load Index (SURG-TLX). Two independent staff urologists reviewed each video using the same assessment tools. Raters’ results were compared with those of the case surgeon. A post-assessment survey and interview were conducted with the case surgeon and the video raters. RESULTS: Over 15 cases, 630 minutes of video was collected of 7 residents and 5 staff surgeons. Six cases were completed by junior residents and 9 by senior residents. The interrater reliability between video assessors was highly correlated (k=0.82), while comparing between intraoperative assessment and video-based assessment, mean O-SCOREs were also highly correlated (r=0.7). When assessing O-SCORE domains for intraoperative performance, there was a slight decrease in correlation between the intraoperative assessment and video-based assessment (r=0.68). All assessors expressed low cognitive load during viewing or while scrubbed into the case with mean SURG-TLX scores of 5 (SD ±1.8). Video-based reviewers frequently rewound the video during critical steps of each case to assess resident performance. CONCLUSIONS: This study showed that video-based assessment of resident performance during ureteroscopy and laser lithotripsy using the O-SCORE is useful and feasible, with high interrater reliability among the case surgeon and independent video reviewers. Use of the SURG-TLX provided new insight into the cognitive load of both case surgeons and video reviewers. Additional research with other types of surgical cases are required to further explore the use of video-based assessment in the OR. Source of Funding: McMaster University Surgical Associates Hamilton, Canada© 2019 by American Urological Association Education and Research, Inc.FiguresReferencesRelatedDetails Volume 201Issue Supplement 4April 2019Page: e513-e513 Advertisement Copyright & Permissions© 2019 by American Urological Association Education and Research, Inc.MetricsAuthor Information Yuding Wang* More articles by this author Kelly Dore More articles by this author Dana Russell More articles by this author Jen Hoogenes More articles by this author Bobby Shayegan More articles by this author Nathan Wong More articles by this author Edward Matsumoto More articles by this author Expand All Advertisement PDF downloadLoading ...
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.005 | 0.023 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.001 |
| Bibliometrics | 0.001 | 0.000 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.001 | 0.001 |
| Open science | 0.001 | 0.001 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.036 | 0.006 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".