A Preliminary Analysis of Different Approaches to Preparing for the USMLE Step 1

Prior to taking the United States Medical Licensing Examination (USMLE) Step 1, medical students commonly spend large amounts of time studying on their own or in groups. They also may participate in test-preparation activities offered by their schools. In addition, there is anecdotal evidence indicating that medical students increasingly purchase “board prep” publications and sign up for commercial coaching courses that may last several weeks and cost thousands of dollars. The effectiveness of alternate approaches to preparing for Step 1 is unknown, though research on other high-stakes exams suggests that exam performance may be improved by coaching courses.1,2,3,4 In this article, we report the results of a recent survey of the strategies used by medical students to prepare for Step 1, and present our preliminary analyses of the relationships between preparation strategies and test scores. Method Participants in this study were chosen by taking a random sample of first-time Step 1 takers from U.S. and Canadian allopathic medical schools. In a survey following the June 1998 paper-and-pencil administration of Step 1, participants were asked about their study habits in relation to: the number of hours spent studying for Step 1 each week; the types of materials they had used when studying; the number of weeks of full-time study; and if applicable, a series of questions about coaching course(s). Survey responses from each participant were matched to his or her USMLE Step 1 scores, Medical College Admission Test (MCAT) scores, and undergraduate science grade-point average (GPA). The survey questionnaires was mailed to a random sample of 3,958 first-time takers of Step 1. A total of 1,650 responded, but because MCAT scores and GPAs were not available for all examinees; only 1,217 were included in the analysis reported here. Initially, information about all the test-preparation materials and courses (methods) was examined descriptively. Then, to evaluate the usefulness of the various preparation methods, ordinary least-squares (OLS) regression analysis was employed. The variables included in the equation fell into four sets. Pre-matriculation Characteristics. MCAT total score (sum of biological sciences, physical sciences, and verbal reasoning scores) and adjusted science GPA.5 Scores were calculated using the following formula: adjusted science GPA = undergraduate science GPA × selectivity Index ÷ 1,000, where the selectivity index is equal to the mean Scholastic Aptitude Test score for students at the undergraduate school attended. This adjustment controls, to a degree, for the variation in grading stringency across undergraduate institutions. Study time. Weeks of full-time study and hours studied per week. Preparation methods. Use of USMLE materials, lecture notes, course syllabi, note-taking service, textbooks, commercial study guides, school materials, group study. Coaching course. A 0/1 dummy code that reflects participation in a coaching course plus an interaction term equal to the product of the dummy code and the MCAT total score. Results Descriptive Information about Preparation Methods. Analysis of the responses to the survey showed that 98% of the respondents had used commercial guides. In contrast, only 70% had used the official USMLE General Instructions, Content Description and Sample Items Booklet, which was provided to all examinees with application materials. Other methods had been used less often: lecture notes (39%), note-taking services (6%), textbooks (44%), course syllabi (21%), preparation materials provided by the school (25%), and group study (25%). The survey responses also indicated that 23% of the respondents had enrolled in a commercial coaching course. When asked about the emphasis of the coaching course, 57% of these respondents reported that it had focused on learning Step 1 content, but also included instruction on test-taking strategies. Twenty-eight percent indicated that the emphasis of their courses had been entirely on learning Step 1 content. Less than 5% reported that their courses had spent a majority of the time on test-taking strategies. Examinees were asked to subjectively rate the value of their courses on a scale of 1 to 5, with 1 being not helpful and 5 being very helpful. The examinees' average rating was 3.2. Analysis of mean scores attained by users versus non-users of certain study methods showed significantly better performances among examinees who had used the USMLE general instructions booklet, textbooks, course syllabi, and study materials provided by the medical school. Examinees who had enrolled in coaching courses received on average lower scores than those who did not enroll (Table 1). However, it should be noted that the examinees who had enrolled in coaching courses had significantly lower MCAT scores (28.8 versus 30.2) and adjusted science GPAs (3.70 versus 3.88). Differences in mean scores between users and non-users of other test preparation methods were smaller and statistically insignificant.TABLE 1: Mean USMLE Step 1 Scores by Test-preparation Approach UsedDescriptive Information about Study Time. In order to examine the effects of preparation time, the examinees were asked how many weeks they had spent studying for the exam full time and how many hours per week they studied during that time. Since these examinees were first-time takers from U.S. medical schools, the vast majority had recently completed their second year of medical school; thus, the number of weeks of full-time study was dependent to some degree on when they had completed their course-work. Analysis indicated that the examinees had studied for an average of 5.8 weeks and for 53 hours per week. Scores were positively correlated with weeks of full-time study. However, Step 1 performance showed little relationship to the number of hours of study per week. Regression Analysis of Factors Influencing Step 1 Scores. A series of regression analyses were run to examine the relationships between Step 1 performance and exam-preparation strategies. The first analysis, the “full model,” included all four sets of predictors described previously: pre-matriculation characteristics, study time, preparation methods, and coaching course participation. Then, for each set, two regression equations were estimated: the first included just the predictors in the set, and the second included all predictors except those in the target set. The R2 for the first equation provides an upper bound on the variance explained by the set; the difference between the R2 for the full model and the R2 for the second model provides a lower bound (the proportion of variance uniquely predicted by the set). The full model explained 33.6% of the variance in Step 1 scores. Table 2 provides the upper and lower bounds on the R2 predicted by each set.TABLE 2: Regression Results Concerning the Relationships between USMLE Step 1 Performance and Four Sets of Predictors*Results indicated that pre-matriculation characteristics accounted for nearly all of the explained variance. Study time explained a small amount of variance, with both weeks of study and hours of study per week having a small positive influence on scores. However, neither coaching courses nor preparation methods had a significant influence on scores. Further, the term measuring the interaction between coaching course enrollment and MCAT scores was not significant. In order to focus on preparation methods in more detail, we examined each individual method separately, controlling for the pre-matriculation characteristics. The results showed that use of textbooks had a significant effect on Step 1 scores, although the effect was fairly small (1.9 points). The other preparation methods did not significantly affect scores. Discussion Studies of the possible effects of coaching courses have been reported for several post-secondary exams, but have been notably absent for the USMLE; the only studies we could locate were undertaken in the 1970s at single medical schools for the NBME Part I exam. In the first of these studies, Scott et al.6 found significantly higher scores in coached examinees in only one of the three years studied. They found that coaching offered greater benefit to students with lower basic science GPAs (first two years of medical school coursework) than it did to students with higher basic science GPAs. Students were surveyed as part of that study, and the vast majority of students thought the course had been beneficial. Both the authors and the students surveyed cited the relevance of the course content to Part I and the organization of the material as the most valuable features of the course. In contrast, Lewis and Kuske7 reported that after controlling for the examinees' basic science GPAs, commercial review courses had no detectable effect on scores. In the present study, the examinees had used several different strategies while preparing for the USMLE Step 1. By far the most common approach was to use commercially prepared study guides. These had been used by 98% of the survey respondents, indeed by more respondents than had used the traditional textbooks they are generally required to buy. The use of commercial study guides also eclipsed the use of materials prepared or approved by professional medical school educators, and eclipsed use of the USMLE publication designed specifically for Step 1 preparation. Ironically, the results obtained indicate that examinees may benefit by using standard texts, many of which they have already purchased. There was little or no evidence of achievement of higher scores as a consequence of using commercially prepared material, controlling for pre-matriculation characteristics and other study methods used. Perhaps the most interesting finding in this analysis is the limited impact of coaching courses on scores. Several caveats must accompany this finding. First, students who enroll in coaching courses are self-selected, in some instances because they are concerned about their readiness to take Step 1. This self-selected group may disproportionately include students in academic trouble at their medical schools and those who have been warned that they are in danger of failing based on tests given in medical school. Second, time-intensive courses may compete with other preparation methods that are more effective or time-efficient. Third, our study lumped together all coaching courses, which vary in length, intensity, and teaching methods. With these provisos, our findings suggest that participation in coaching courses appears to have little effect on scores when controlling for educational antecedents, time of study, and other preparation methods. Last, the reduction in sample size due to survey return rate and incomplete data was problematic. Examinees included in the study had slightly higher Step 1 scores (220.7 versus 216.1). Thus, there is some evidence to support the intuitive reasonableness of selection bias.

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.