Use of an Objective Structured Assessment of Technical Skill (OSATS) after a Sports Rotation
Bibliographic record
Abstract
Objectives: While the acquisition of competent technical skill is a defining characteristic of surgery, it is not measured systematically in residency. As all post-graduate medical training slowly shifts to a competency-based model, effective assessments of surgical and other technical skills after modules will become necessary. However, the best method for of assessing competence in technical skill in this setting is unknown, and is limited by both cost and access to resources. We hypothesized that a multi-station Objective Structured Assessment of Technical skill (OSATS), using sawbones models, would be a valid and reliable method of assessing resident competence in surgical skills after a sports medicine rotation. Methods: At the start of their three-month sports medicine rotation, each resident was provided a list of 10 surgical skills in which they were expected to demonstrate competence (Table 1). At the end of the rotation, each resident undertook an OSATS comprised of six randomly chosen stations - low-fidelity sawbones models were used in all stations. Residents were evaluated by faculty / staff surgeons using a previously validated global rating scale (the Arthroscopic Surgical Skill Evaluation Tool (ASSET)), as well as task-specific checklists created using a modified Delphi procedure, and a final five-point global rating scale (GRS) using the Drefus model of skill acquisition (1=novice, 2=advanced beginner, 3=competent, 4=proficient, 5=expert). All arthroscopic procedures were recorded, and all hand movements were videotaped - the videos were reviewed by a single, blinded observer, and correlation sought between the faculty ratings and the observer ratings. Results: Over 18 months, 27 residents (19 junior, 8 senior) sat the OSATS after their rotation, as well as seven sports medicine staff and seven fellows, for a total of 41 participants. The overall reliability of the OSATS as measured by Cronbach's Alpha was very high (0.9). A significant difference by year in training was seen for the overall GRS, the total ASSET score, and the total checklist score, as well as for each individual technical skill (p<0.001) - this difference was also seen for all stations. Post hoc analysis demonstrated a significant difference in the total ASSET score between junior (PGY1-3) and senior (PGY4&5) residents, senior residents and fellows, as well as between fellows and faculty (p<0.05)(Figure 1). A high correlation was seen between the faculty assessments and the blinded observer assessments for each station (>0.8). Conclusion: The results of this study demonstrate that an OSATS using dry models is a valid and reliable means of assessing technical skill in orthopaedic residents after a sports medical rotation. Interestingly, junior residents were not able to perform technical skills as well as senior residents despite an identical rotation, suggesting that overall surgical experience and exposure is as important as intensive teaching.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.003 | 0.007 |
| Meta-epidemiology (narrow) | 0.001 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.001 |
| Bibliometrics | 0.001 | 0.000 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.001 |
| Open science | 0.000 | 0.001 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.002 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".