Board 390 - Research Abstract Assessment of the Challenging CanMEDS Competencies

Introduction/Background The Royal College of Physicians and Surgeons of Canada developed the Canadian Medical Directives for Specialist (CanMEDS) with seven core competencies: Medical Expert (ME) and six Intrinsic competencies Communicator, Collaborator, Manager, Health Advocate, Scholar and Professional. Competency of Professional, Health Advocate and Scholar (PHAS) CanMEDS competencies are difficult to define and assess during clinical practice and in simulations, in contrast to Medical Expert (ME) and other Intrinsic competencies. Our objective was to collect evidence to support construct validity of revised Generic Integrated Objective Structured Assessment Tool (GIOSAT) including content, response process, internal structure, relation to other variables and consequences using simulated scenarios targeting PHAS competencies. Research Question: Can we collect evidence to support construct validity for Professional, Health Advocate and Scholar CanMEDS competencies assessment Results for anesthesia residents performing two simulation scenarios using the Generic Integrated Objective Structured Assessment Tool and four trained blinded raters? Methods REB approval and informed consent was obtained for a prospective single blind correlation study. Twenty one anesthesia residents rotating at the University of Ottawa volunteered in this study where each of them performed both scenarios as the primary physician to manage the situation. Content: Two simulation scenarios: Do-not resuscitate (DNR) and Morphine overdose (MOD) with disclosure, were developed by a panel of experts highlighting PHAS competencies.1-3 GIOSAT is divided in two sections ME with eight items and Intrinsic with six items. Each item has abbreviated anchors and is scored with a Likert rating scale (1=very poor to 6=very good). Response process: Pilot scenarios performed by actors at optimal and sub-optimal level of performance were used to train four the raters from different institution blinded from residents identity. Raters rules were created to define borderline performances. Twenty one anesthesia residents volunteered to participate as primary physicians to manage the simulation scenarios. Internal structure was analyzed with inter-rater intra-class correlations (ICCs) and generalizability studies for ME and intrinsic and also for ME and PHAS. Relation with other variables: Comparison between scenario scores was performed with Student’s t -test. Our primary outcome was the correlations between post-graduate year of residency (PGY) and average PHAS, Intrinsic, Medical Expert and Total scores. The secondary outcome was the correlation between PHAS scores with Intrinsic, ME and total GIOSAT scores. Results ICCs for PHAS, Intrinsic, Medical Expert (ME) and total scores single measures were moderate in both scenarios (.42-.68, p<.000), and for average scores (were substantial to almost perfect (.76-.88. p<.000). Participant (p) accounted for 23% of variance and 20% for PHAS. Scenario (s) and raters (r) did not account for important variation component (VC) but the interaction between ps and psr accounted for 14 and 19 %VC for ME and Intrinsic respectively. G-study for PHAS had similar Results with ps accounting for 7% VC and psr accounting for 17% VC. (Table 1) G-coefficient for the Intrinsic was .64 and .66 for PHAS. Two raters and eight scenarios using ME and Intrinsic are required to obtain a G-coefficient >.8. Two raters and eleven scenarios using ME and PHAS are required to obtain a G-coefficient >.8.(Table 2) PGY correlated with PHAS (r=.59, p=.004), Intrinsic (r= .65, p=.002) and total scores (r=.46, p=.034) but not with ME (r= .26, p=.25). PHAS scores significantly correlated with and Intrinsic (r=.98, p<.000), ME (r= .7, p<.001) and Total (r=.89, p<.000). Conclusion Our study demonstrates construct validity evidence for assessing PHAS and Intrinsic competencies using clinical simulation with a G-coefficient of .64. Future studies with similar methodology may support construct validity at high stakes level using two raters and eight or more scenarios. References 1. Frank, J. (2005). The CanMEDS 2005 Physician Competency Framework Edited by Royal College of Physicians and Surgeons od Canada. Available from URL: http://www.rcpsc.medical.org. 2. Neira, V. M., Bould, M. D., Nakajima, A., Boet, S., Barrowman, N., Mossdorf, P., … Hamstra, S. J. (2013). “GIOSAT”: a tool to assess CanMEDS competencies during simulated crises. Canadian journal of anaesthesia. 3. Lynch, D. C., Surdyk, P. M., & Eiser, A. R. (2004). Assessing professionalism: a review of the literature. Medical teacher, 26(4), 366–73. 4. Ponton-Carss, A., Hutchison, C., & Violato, C. (2011a). Assessment of communication, professionalism, and surgical skills in an objective structured performance-related examination (OSPRE): a psychometric study. American journal of surgery, 202(4), 433–40. 5. Morrison, L. J., Kierzek, G., Diekema, D. S., Sayre, M. R., Silvers, S. M., Idris, A. H., & Mancini, M. E. (2010). Part 3: ethics: 2010 American Heart Association Guidelines for Cardiopulmonary Resuscitation and Emergency Cardiovascular Care. Circulation, 122(18 Suppl 3), S665–75. 6. Syed, S., Paul, J. E., Hueftlein, M., Kampf, M., & McLean, R. F. (2006). Morphine overdose from error propagation on an acute pain service. Canadian journal of anaesthesia. 7. The Canadian Medical Protective Association. (2008). Communicating with your patient about harm DISCLOSURE ROAD MAP. Retrieved from www.cmpa-acpm.ca. Disclosures None.

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.