Incorporating Situational Judgment Tests Into Postgraduate Medical Education Admissions: Examining Educational and Organizational Outcomes

Purpose: CASPer is an online situational judgment test (SJT) that has been developed for use in medical school admissions, with a separate version developed for admission into specialty training. CASPer was developed to be a broad measure of personal and professional qualities for the entire applicant pool at the time of screening to help bring better quality applicants to interview. The purpose of this project was to examine if using CASPer in the residency selection process impacted the prevalence and type of professionalism issues, formal remediation incidents, and associated cost savings within the college. Methods: Resident in difficulty documentation (type of intervention, CanMEDs areas of difficulty, professionalism issues, and costs) across 4 years before the implementation of CASPer (pre-CASPer cohort) and 4 years post-CASPer implementation (post-CASPer cohort) were reviewed. Descriptive statistics and between-group comparisons were used to explore type of interventions and associated problems. Professionalism issues, as documented in resident files, were categorized into different types of unprofessional behavior based on frameworks proposed by Mak-van der Vossen et al 1 and Hilton and Stolnick. 2 Results: The number of residents identified to be in difficulty during the pre- and post-CASPer time frames were similar (16 and 15 residents, respectively). Likewise, the number of interventions within each cohort were comparable, with 18 interventions documented in the pre-CASPer cohort and 16 interventions documented in the post-CASPer cohort. Despite these similarities, the number of residents requiring formal learning interventions (i.e., remediation or probation) in the pre-CASPer group were significantly higher (P < .05) when compared with the post-CASPer cohort (15 vs 5 respectively). The number of residents requiring informal learning interventions (i.e., enhanced learning plans) that allow the residents to continue the program with additional focused effort in areas that need to be addressed increased from 3 (pre-CASPer cohort) to 11 (post-CASPer cohort). The reduction in formal learning interventions from the pre- to post-CASPer group was associated with a 96% reduction in costs (e.g., salary for additional training, preceptor remunerations, additional assessments to tailor interventions, logistics [vacations, leaves, travel], resident resource office support), from hundreds of thousands to tens of thousands of dollars spent in resources. Within these formal and informal interventions, the medical expert domain was found to be the most frequent role requiring attention in both the pre- (16/16,100%) and post-CASPer cohorts (12/15, 80%). Professionalism issues were identified in 75% of pre-CASPer cases but were found in reduced frequency in the post-CASPer group (40%). Categorization of the professionalism issues showed an overall reduction in professionalism concerns, from pre- to post-CASPer cohorts, across all domains (e.g., involvement, integrity, interaction, introspection, ethical practice, reflection/self-awareness, responsibility/accountability, respect for patients, social responsibility) except teamwork. Discussion: The results of this study suggest that the inclusion of the SJT CASPer in the screening of applicants to postgraduate medical training provides important information that can result in a reduction in the number of formal interventions and number of professionalism concerns among selected residents, subsequently reducing associated costs as well as faculty and staff time. Significance: In addition to the immediate benefits of integrating SJTs in the applicant selection process, the cost savings associated with reduced formal interventions can be redirected to enhancing institutional endeavors (e.g., Competence by Design launch) and improving programs (e.g., additional funding for courses and well-being work). Acknowledgments: The National Board of Medical Examiners Stemmler Fund for their original support of CASPer creation.

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.