Development and evaluation of the General Surgery Objective Structured Assessment of Technical Skill (GOSATS)
Bibliographic record
Abstract
Abstract Background Technical skill acquisition is important in surgery specialty training. Despite an emphasis on competency-based training, few tools are currently available for direct technical skills assessment at the completion of training. The aim of this study was to develop and validate a simulated technical skill examination for graduating (postgraduate year (PGY)5) general surgery trainees. Methods A simulated eight-station, procedure-based general surgery technical skills examination was developed. Board-certified general surgeons blinded to the level of training rated performance of PGY3 and PGY5 trainees by means of validated scoring. Cronbach's α was used to calculate reliability indices, and a conjunctive model to set a pass score with borderline regression methodology. Subkoviak methodology was employed to assess the reliability of the pass–fail decision. The relationship between passing the examination and PGY level was evaluated using χ2 analysis. Results Ten PGY3 and nine PGY5 trainees were included. Interstation reliability was 0·66, and inter-rater reliability for three stations was 0·92, 0·97 and 0·76. A pass score of 176·8 of 280 (63·1 per cent) was set. The pass rate for PGY5 trainees was 78 per cent (7 of 9), compared with 30 per cent (3 of 10) for PGY3 trainees. Reliability of the pass–fail decision had an agreement coefficient of 0·88. Graduating trainees were significantly more likely to pass the examination than PGY3 trainees (χ2 = 4·34, P = 0·037). Conclusion A summative general surgery technical skills examination was developed with reliability indices within the range needed for high-stakes assessments. Further evaluation is required before the examination can be used in decisions regarding certification.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.012 | 0.021 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.001 |
| Bibliometrics | 0.001 | 0.000 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.001 | 0.000 |
| Open science | 0.001 | 0.001 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.001 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".