Board 421 - Research Abstract A Systematic Approach to Design Clinical Performance Checklists (Submission #227)
Bibliographic record
Abstract
Introduction/Background Assessing performance helps to identify the abilities of clinicians and potential performance gaps, augments debriefings and is essential for scientific studies investigating factors influencing clinical performance.1 Checklists are widespread tools to assess performance. But while the development process of such a performance checklist is essential for its quality, existing studies rarely describe their checklist development in detail. Current methodological recommendations2,3 fail to provide a systematic, step-by-step approach to develop clinical performance checklists. Such a systematic approach would support researchers in evaluating the suitability of the checklists for different contexts, designing performance assessment tools for specific clinical scenarios reflecting precisely the task demands on the clinician and in either adapting existing checklists or generating new ones. Thus, the aim of this study was to provide an overall systematic approach to develop clinical performance checklists. Using the example of a simulated sepsis scenario we illustrate our five development steps. Methods Step 1 - Draft Checklist: Based on the literature and own clinical experience we designed a draft checklist. Step 2 – The Delphi-Review-Rounds: We sent out the draft checklist to five experts for reviewing using an adapted Delphi-Method.4 Step 3 – Design of the final checklist and pilot testing: Every checklist item was then divided into three scoring categories: task not performed (0 points); task performed partially (1 point), and task performed completely (2 points). Then the checklist was tested by rating video clips of simulation trainings and a few adjustments have been made. This step is indispensable; by applying the checklist to a set of different examples, the raters experience the applicability of the items and the usability of the rating scale. Step 4 – Final Delphi-Review-Round: To assure that the changes made after the pilot testing are generally valid the checklist resulted from step three was sent out again to the five experts. Step 5 – Items weighting: In the last step we sent out the checklist to 30 pediatricians and instructed them to rate all actions in terms of their importance for the success of the treatment. The mean importance score serves as a weighting factor for every item. This way we get a more accurate assessment of performance because the checklist differentiates more between essential and less important items. Validity testing – Six videos of septic shock simulation training were independently rated from two raters. Interrater reliability was calculated using Cronbach’s α; criterion validity was tested by investigating the relationship between the checklist score and three external criterion: team experience level, experience level of the leader and a global performance rating (rating from 1-10). Results We successfully applied our five step approach and we developed a performance checklist including 33 items for a simulated paediatric sepsis scenario. Cronbach’s α ranged from acceptable (αα = .6) to very good (α = .9). Criterion validity is given: Significant correlation between the checklist score and i) mean experience level of team (r = .37, p= .05) ii) leader experience level (r = .44, p= .05) iii) global performance rating score (r = .54, p= .05). Conclusion We described a systematic approach to design clinical performance checklists that integrates the published evidence and the knowledge of domain experts. The validity of the checklist has been confirmed. A structured development process is a necessary prerequisite of a valid checklist. Only if a widely recognized standard for developing performance checklists is established we can design appropriate measurement tools and move the field of performance assessment in healthcare forward. References 1. Boulet JR, Murray D. Review article: Assessment in anesthesiology education. Canadian Journal of Anesthesia/Journal canadien d’anesthésie. 2011:1-11. 2. Stufflebeam DL. Guidelines for developing evaluation checklists: the checklists development checklist (CDC). [monograph on the Internet]. 2000; http://www.wmich.edu/evalctr/archive_checklists/guidelines_cdc.pdf. Accessed Dezember 17, 2012. 3. Scriven M. The logic and methodology of checklists. Retrieved on. 2000;11:02-07. 4. Clayton MJ. Delphi: a technique to harness expert opinion for critical decision†making tasks in education. Educational Psychology. 1997;17(4):373-386. Disclosures Salary Support from Center for Medical Simulation to teach on simulation courses none Per dien honoraria from PAEDSIM e.V. to teach on pediatric simulation courses non-profit organization PAEDSIM e.V.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.013 | 0.002 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.000 | 0.001 |
| Science and technology studies | 0.001 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.001 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".