Towards Robust Validity Evidence for Learning Environment Assessment Tools

To the Editor: Colbert-Getz and colleagues’1 review of learning environment (LE) assessment tools is both timely and relevant. The authors use four of the five categories of validity evidence from the American Psychological and Educational Research Associations to arrive at a total validity evidence score for each tool they reviewed to judge the quality of both medical student and resident LE perceptions. We wonder, however, if the authors used only the initial publications to arrive at their total validity evidence scores. For example, while the Medical Student Learning Environment Survey’s (MSLES) original publication received a total score of 3/8 (38%), subsequent publications from Australia and Canada examined the Internal Structure and Relationship to Other Variables criteria.2,3 Both studies used factor analysis to independently confirm that the individual scales of the MSLES represent one dominant factor. Clarke et al2 also examined the retest reliability and internal consistency, while Rusticus et al3 correlated the MSLES to student satisfaction and academic performance. Applying the authors’ validity criteria, we would have given an additional rating of 2 (“strong” evidence) for Internal Structure and a score of 1 (“weak” evidence) for Relationship to Other Variables. The total validity evidence score of the MSLES would therefore increase to 6/8 (75%). Similarly, for measuring the resident LE, the authors give the VA Learners’ Perception Survey (LPS) a total validity score of 2/8 (25%). The original publication by Keitz et al4 used focus groups of medical students and residents in the initial development of the LPS, and factor analysis was used to collapse the original 57 questions into four major domains. Internal consistency using a mixed-effects model was further verified in a subsequent publication by Cannon et al.5 We would have given an additional rating of 1 for Response Process and 2 for Internal Structure, increasing the total validity evidence score to 5/8 (63%). Both of these updated scores for the MSLES and LPS would be the highest scores listed for validity evidence in undergraduate and graduate medical education, respectively. We also wonder if the authors assessed the interrater reliability used to assess the validity evidence, since their checklist was adapted from Beckman et al,6 which found kappa values ranging from −0.10 to 0.96 and was particularly poor for rating the Response Process criteria. Lawrence K. Loo, MD Vice chair, Education and Faculty Development, Department of Medicine, and professor of medicine, Loma Linda University School of Medicine, Loma Linda, California; [email protected] John M. Byrne, DO Associate chief of staff, Education, VA Loma Linda Healthcare System, and associate professor of medicine, Loma Linda University School of Medicine, Loma Linda, California.

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.