Using Kane’s validity framework to examine the implications of feedback in simulation-based assessments
Bibliographic record
Abstract
BACKGROUND: Simulation modalities have been increasingly used within programmatic assessment systems, yet educators typically have not collected and appraised validity evidence to justify such uses. Kane's validity framework offers a contemporary approach to conducting validation studies of assessment practices. Under the framework, educators collect and appraise validity evidence according to four inferences: the scoring of performance, the generalization of scores to other assessment contexts, the extrapolation of assessment performance to real-world contexts, and the implications or consequences of assessment decisions for learners, educators, programs, patients, and society. We developed a simulation-based echocardiography competence assessment tool (ECAT) and collected validity evidence to evaluate its use as an assessment for learning. We applied Kane's validity framework to evaluate the utility of the ECAT, with a focus on the implications of the assessment for promoting trainees' learning. METHODS: We implemented the ECAT in 2017, collecting simulation-based performance data and subsequent interview data. Fourteen cardiology trainees were assessed using the ECAT by four raters, and their performance was video-recorded. After trainees reviewed their performance videos and feedback, we conducted individual interviews with them and the raters who provided feedback. Directed content analysis generated implications and scoring evidence, and quantitative analyses generated scoring and extrapolation evidence. All evidence was critically appraised to form a validity argument about using ECAT as an assessment for learning. RESULTS: Participants reported that ECAT scores accurately represented trainees'performance, and that the feedback helped identify learning opportunities. Inter-rater reliability was high at ICC = 0.913 (95% CI 0.81-0.97). Participants' ECAT scores correlated with their end-of-rotation cardiology exam scores (r = 0.66, p = 0.02) and had positive associations with raters' judgments of the diagnostic quality of their scans, and with their reported numbers of echocardiograms seen, performed, and interpreted. CONCLUSIONS: Our integrated analysis produced a data-informed validity argument supporting the use of the ECAT as a simulation-based assessment for learning. The findings also highlighted multiple areas for further research to optimize the ECAT. Our illustrative example of Kane's validity framework aims to support simulation educators as they are increasingly called on to justify the use of simulation-based assessments in programmatic and competency-based assessment systems.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.590 | 0.811 |
| Meta-epidemiology (narrow) | 0.002 | 0.002 |
| Meta-epidemiology (broad) | 0.004 | 0.008 |
| Bibliometrics | 0.022 | 0.011 |
| Science and technology studies | 0.007 | 0.039 |
| Scholarly communication | 0.012 | 0.016 |
| Open science | 0.005 | 0.014 |
| Research integrity | 0.005 | 0.007 |
| Insufficient payload (model declined to judge) | 0.003 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".