Simulation-based Assessment of Paramedics and Performance in Real Clinical Contexts
Bibliographic record
Abstract
OBJECTIVE: The objective of this study was to seek validity evidence for simulation-based assessments (SBA) of paramedics by asking to what extent the measurements obtained in SBA of clinical competence are associated with measurements obtained in actual paramedic contexts, with real patients. METHODS: This prospective observational study involved analyzing the assessment of paramedic trainees at the entry-to-practice level in both simulation- and workplace-based settings. The SBA followed an OSCE structure involving full clinical cases from initial patient contact to transport or transfer of care. The workplace-based assessment (WBA) involved rating samples of clinical performance during real clinical encounters while assigned to an emergency medical service. For each candidate, both assessments were completed during a 3-week period at the end of their training. Raters in the SBA and WBA settings used the same paramedic-specific seven-dimension global rating scale. Reliability was calculated and decision studies were completed using generalizability theory. Associations between settings (overall and by dimension) were calculated using Pearson's correlation. RESULTS: A total of 49 paramedic trainees were assessed using both a SBA and WBA. The mean score in the SBA and WBA settings were 4.88 (SD = 0.68) and 5.39 (SD = 0.48), respectively, out of a possible 7. Reliability for the SBA and WBA settings reached 0.55 and 0.49, respectively. A decision study revealed 10 and 13 cases would be needed to reach a reliability of 0.7 for the SBA and WBA settings. Pearson correlation reached 0.37 (p = 0.01) between settings, which rose to 0.73 when controlling for imperfect reliability; five of seven dimensions (situation awareness, history gathering, patient assessment, decision making, and communication) reaching significance. Two dimensions (resource utilization and procedural skills) did not reach significance. CONCLUSION: For five of the seven dimensions believed to represent the construct of paramedic clinical performance, scores obtained in the SBA were associated with scores obtained in real clinical contexts with real patients. As SBAs are often used to infer clinical competence and predict future clinical performance, this study contributes validity evidence to support these claims as long as the importance of sampling performance broadly and extensively is appreciated and implemented.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.007 | 0.054 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.001 | 0.000 |
| Science and technology studies | 0.000 | 0.001 |
| Scholarly communication | 0.001 | 0.001 |
| Open science | 0.001 | 0.002 |
| Research integrity | 0.000 | 0.001 |
| Insufficient payload (model declined to judge) | 0.002 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".