Board 421 - Research Abstract A Systematic Approach to Design Clinical Performance Checklists (Submission #227)
Notice bibliographique
Résumé
Introduction/Background Assessing performance helps to identify the abilities of clinicians and potential performance gaps, augments debriefings and is essential for scientific studies investigating factors influencing clinical performance.1 Checklists are widespread tools to assess performance. But while the development process of such a performance checklist is essential for its quality, existing studies rarely describe their checklist development in detail. Current methodological recommendations2,3 fail to provide a systematic, step-by-step approach to develop clinical performance checklists. Such a systematic approach would support researchers in evaluating the suitability of the checklists for different contexts, designing performance assessment tools for specific clinical scenarios reflecting precisely the task demands on the clinician and in either adapting existing checklists or generating new ones. Thus, the aim of this study was to provide an overall systematic approach to develop clinical performance checklists. Using the example of a simulated sepsis scenario we illustrate our five development steps. Methods Step 1 - Draft Checklist: Based on the literature and own clinical experience we designed a draft checklist. Step 2 – The Delphi-Review-Rounds: We sent out the draft checklist to five experts for reviewing using an adapted Delphi-Method.4 Step 3 – Design of the final checklist and pilot testing: Every checklist item was then divided into three scoring categories: task not performed (0 points); task performed partially (1 point), and task performed completely (2 points). Then the checklist was tested by rating video clips of simulation trainings and a few adjustments have been made. This step is indispensable; by applying the checklist to a set of different examples, the raters experience the applicability of the items and the usability of the rating scale. Step 4 – Final Delphi-Review-Round: To assure that the changes made after the pilot testing are generally valid the checklist resulted from step three was sent out again to the five experts. Step 5 – Items weighting: In the last step we sent out the checklist to 30 pediatricians and instructed them to rate all actions in terms of their importance for the success of the treatment. The mean importance score serves as a weighting factor for every item. This way we get a more accurate assessment of performance because the checklist differentiates more between essential and less important items. Validity testing – Six videos of septic shock simulation training were independently rated from two raters. Interrater reliability was calculated using Cronbach’s α; criterion validity was tested by investigating the relationship between the checklist score and three external criterion: team experience level, experience level of the leader and a global performance rating (rating from 1-10). Results We successfully applied our five step approach and we developed a performance checklist including 33 items for a simulated paediatric sepsis scenario. Cronbach’s α ranged from acceptable (αα = .6) to very good (α = .9). Criterion validity is given: Significant correlation between the checklist score and i) mean experience level of team (r = .37, p= .05) ii) leader experience level (r = .44, p= .05) iii) global performance rating score (r = .54, p= .05). Conclusion We described a systematic approach to design clinical performance checklists that integrates the published evidence and the knowledge of domain experts. The validity of the checklist has been confirmed. A structured development process is a necessary prerequisite of a valid checklist. Only if a widely recognized standard for developing performance checklists is established we can design appropriate measurement tools and move the field of performance assessment in healthcare forward. References 1. Boulet JR, Murray D. Review article: Assessment in anesthesiology education. Canadian Journal of Anesthesia/Journal canadien d’anesthésie. 2011:1-11. 2. Stufflebeam DL. Guidelines for developing evaluation checklists: the checklists development checklist (CDC). [monograph on the Internet]. 2000; http://www.wmich.edu/evalctr/archive_checklists/guidelines_cdc.pdf. Accessed Dezember 17, 2012. 3. Scriven M. The logic and methodology of checklists. Retrieved on. 2000;11:02-07. 4. Clayton MJ. Delphi: a technique to harness expert opinion for critical decision†making tasks in education. Educational Psychology. 1997;17(4):373-386. Disclosures Salary Support from Center for Medical Simulation to teach on simulation courses none Per dien honoraria from PAEDSIM e.V. to teach on pediatric simulation courses non-profit organization PAEDSIM e.V.
Récupéré en direct depuis OpenAlex et désinversé. Les résumés ne sont pas conservés dans cette base de données : les index inversés représentent 8,6 Go des 9,3 Go de texte de la base, et le serveur dispose de 13 Go libres.
Comment cette classification a été obtenuedéplier
Prédiction distillée sur la base complète
Imitation des enseignantsNi prévalence calibrée, ni vérité terrain. Validation humaine à venir. Apprise à partir de 10 348 étiquettes directes de Codex et de 10 348 étiquettes directes de Gemma. Le mode candidate est l'union des têtes enseignantes seuillées; le consensus est leur intersection. Ces sorties portent le statut machine_predicted_unvalidated et ne sont ni des étiquettes humaines ni des étiquettes directes de modèles de pointe.
Scores Codex et Gemma par catégorie
| Catégorie | Codex | Gemma |
|---|---|---|
| Métarecherche | 0,013 | 0,002 |
| Méta-épidémiologie (sens strict) | 0,000 | 0,000 |
| Méta-épidémiologie (sens large) | 0,001 | 0,001 |
| Bibliométrie | 0,000 | 0,001 |
| Études des sciences et des technologies | 0,001 | 0,000 |
| Communication savante | 0,000 | 0,000 |
| Science ouverte | 0,000 | 0,000 |
| Intégrité de la recherche | 0,000 | 0,001 |
| Charge utile insuffisante (le modèle a refusé de juger) | 0,000 | 0,000 |
Scores machine (provisoires)
Les deux têtes enseignantes du modèle étudiant, lues sur ce travail. Un score ordonne la base pour la relecture; il n'affirme jamais une catégorie, et le statut de validation accompagne chaque rangée tel quel.
Scores de référence d'un modèle non mature (critères de maturité non atteints, 7 itérations). Un score ordonne; il n'affirme jamais une catégorie.
score_only:v0-immature-baseline · tel quel depuis la passe de notation : score_only signifie que le nombre peut ordonner les travaux, et qu'aucune étiquette de catégorie n'en découleClassification
machine, non validéePrédiction automatique; un appel candidat d’une seule tête enseignante, pas un consensus.
Le détail, modèle par modèle et score par score, se trouve en fin de page sous « Comment cette classification a été obtenue ».