Instructions for Obtaining Journal CME Credit

Simulation in Healthcare's journal-based CME program is open to all readers. Members of the Society for Simulation in Healthcare participate at a preferred rate, but you need not be a SSH member or a journal subscriber to take part in this CME activity. Please complete the following steps: Read the article entitled A Comparison of Global Rating Scale and Checklist Scores in the Validation of an Evaluation Tool to Assess Performance in the Resuscitation of Critically Ill Patients During Simulated Emergencies (abbreviated as “CRM Simulation Study 1B”) by John Kim, MD, MEd, FRCPC, Department of Critical Care Medicine, University of Ottawa/The Ottawa Hospital, Ottawa, Canada. Review the questions and other required information for the CME program completion. When ready, go to the CME website: www.ssih.org. Submit your answers, form of payment and other required information by December 31 of the year following the year of publication. The Society for Simulation in Healthcare is approved by the Accreditation Council for Continuing Medical Education (ACCME) to sponsor continuing medical education programs for physicians. The Society for Simulation in Healthcare designates this educational activity for a maximum of 1 AMA PRA Category 1 Credit™. Physicians should only claim credit commensurate with the extent of their participating in the activity. Purpose: The focus of the journal-based CME program and the articles chosen for the program is to educate readers on current developments in simulation education, technology, and techniques. Target Audience: Physicians and other healthcare professionals who are involved with simulation in the healthcare field. Learning Objective: After reading the featured article, participants will be able to identify the components required to formally validate rating instruments in high-stakes evaluation of performance and to identify the key issues in validation with the Ottawa GRS and Ottawa CRM checklist. Disclosure Information Question authors: John Kim, MD, MEd, FRCPC. The question authors have indicated that they have nothing to disclose. Question 1 In the study examining the use of the Ottawa Global Rating Scale (GRS) and Ottawa Crisis Resource Management (CRM) checklist as potential instruments for the evaluation of CRM skills, all of the following components of construct validity were examined except for: Content validity Response process Relationship to other variables Comparison to gold standard testing Critique In this study, multiple components of construct validity were examined, in keeping with the American Educational Research Association/American Psychologic Association Standards for Educational and Psychologic Assessment.1 A Delphi peer-review process of the Ottawa CRM GRS and the Ottawa CRM Checklist established content validity. Response process was established by a standardization of case administration and a Delphi peer-review process to ensure uniformity of test scoring. This study examined the relationship to the variable of residency training by comparing scores in first and third-year postgraduate (PGY-1 and PGY-3) participants. Interrater and interitem reliability of both instruments were examined to establish whether internal structure was present. Because there is no formal method for the evaluation of CRM performance, consequential validity could not be examined. Therefore, the answer to question 1 is d. Question 2 Study enrollment was conducted by recruitment of first and third-year postgraduate (PGY-1 and PGY-3) residents based on the following hypotheses: PGY-3 residents have received more formal training in CRM than PGY-1 residents, and therefore should score higher on the Ottawa GRS/Ottawa CRM checklists. PGY-3 residents have more simulation training than PGY-1 residents, and therefore should score higher on the Ottawa GRS/Ottawa CRM checklists. PGY-3 residents have more clinical experience than PGY-1 residents, and therefore should score higher on the Ottawa GRS/Ottawa CRM checklists. All of the above. Critique Because there is no proven method to evaluate CRM performance, identifying groups at different levels of CRM performance and examining a rating instrument's ability to differentiate between both groups is essential.1,2 The only certain difference between PGY-1 and PGY-3 residents is the amount of clinical experience for each group. CRM is not formally taught in most undergraduate and postgraduate training programs, therefore any discernable difference in CRM training exists for each group. Furthermore, the study's protocol for enrollment excluded all residents with prior simulator experience, therefore removing the variable of simulator experience from influencing performance. Question 3 In comparing the Ottawa GRS and the Ottawa CRM checklist, what conclusions can be drawn? The Ottawa GRS and Ottawa CRM checklist demonstrate multiple properties to suggest that construct validity is present with both instruments. The Ottawa GRS seems to be more feasible to use than the Ottawa CRM checklist. The Ottawa GRS seems superior for use in formative evaluation as residents progress in training, given its wider range of scored performance. The Ottawa CRM checklist seems superior for us in formative evaluation as residents progress in training, given the multiple items being scored. Critique The Ottawa GRS and Ottawa CRM checklist both seem to provide evidence for construct validity to be present. Both demonstrate the presence of content validity, response process, and response the variable of resident training. Both also demonstrate issues in internal structure, specifically in the area of interrater and interitem reliability. These issues will need to be addressed before either instrument can be used in high-stakes evaluation of performance. Feasibility was not studied as a part of the original study design. Although the evaluators all demonstrated a preference for the Ottawa GRS in rating performance, this is not sufficient to establish a superior choice. Similarly, neither instrument was evaluated for its ability to provide formative feedback. The Ottawa GRS may prove more useful in this aspect by its wider range of performance, whereas the Ottawa CRM checklist may be preferred because of the number of items being scored. The present study does not address this specific issue. Question 4 To complete the formal validation of the Ottawa GRS and/or Ottawa CRM checklist in the evaluation of CRM during medical emergencies, all of the following steps will be required except: Revision of the rater training process Revision of the instrument design process Comparison of performance of study participants as they progress in residency training Comparison of Ottawa GRS and CRM checklist scores to gold standard evaluations of performance Critique The Ottawa GRS and Ottawa CRM checklist displayed evidence to support the presence of content validity, response process, and response to the level of training. However, one key weakness was the presence of only moderate interrater reliability in overall scores, and poor interrater rater reliability in two of the five categories of CRM being tested. In order for high-stakes evaluation to occur with any instrument, higher levels of interrater reliability must be present. Therefore, revisions to both the rater training process and instrument design must be undertaken. In addition, although the study identified differences between groups with different levels of training, the ability of the instruments to detect differences in individuals as their skill levels improve would provide even more evidence to support its use in high-stakes evaluation. Unfortunately, as there exists no established method to formally evaluate CRM performance, no gold standard comparison can take place.

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.