Spaced Repetition in a Cohort of Practicing Physicians: Methods and Preliminary Results
Bibliographic record
Abstract
Spaced repeated testing over time results in better long-term knowledge retention than repeated study of the same material. It is particularly effective when feedback is provided, initial repetitions occur early, and answering questions requires application of knowledge through use of short-answer or context-rich multiple-choice questions. 1–4 American Board of Medical Specialties boards incorporate longitudinal knowledge assessments in their continuing certification programs, 5 but most have not yet systematically incorporated spaced repetition. The goal of this study is to compare the effect of 5 different spaced repetition strategies on American Board of Family Medicine (ABFM) Diplomate knowledge retention and transfer of knowledge from one clinical scenario to another. We report the study methods and will report preliminary results from this ongoing work. Physicians participating in the ABFM Continuing Knowledge Self-Assessment (CKSA) receive 25 questions per quarter. After answering each question, they rate their confidence in their answer on a 6-point scale before receiving feedback, an educational critique, and list of appropriate references. Twenty-six thousand two hundred fifty-five family physicians who completed the CKSA in the fourth quarter of 2020 (baseline period) were eligible for study inclusion. Participants were randomized to a control group or 1 of 5 spaced repetition conditions over the subsequent 5 calendar quarters (January 1, 2021, to March 31, 2022). Control group participants received no repeated questions during this time period. Participants in the other 5 groups received 6 questions repeated either once or twice; the interval between repetitions differed between each group. Incorrectly answered baseline questions were prioritized for spaced repetition—those answered extremely confidently received highest priority, with decreasing priority for those questions answered with lesser degrees of confidence. If necessary, correctly answered questions could then be selected for spaced repetition, with higher priority for questions not answered confidently (e.g., “guesses”) than those answered more confidently. Physicians remain in the study unless opting out or failing to complete all 25 CKSA questions in a quarter in which they were scheduled to receive repeated questions. In quarter 6 of the study, all remaining participants will receive their 6 repeated questions. The primary analysis will compare differences in enduring learning (the percentage of incorrectly answered baseline questions subsequently answered correctly during quarter 6) between physicians receiving any spaced repetition questions in quarters 1–5 compared with the control group, who will have gone 18 months without seeing repeated questions. Subgroup analyses will compare differences in learning between physicians receiving one repetition compared with those receiving 2, differences in learning between the 2 single-spaced repletion strategies, and differences in learning between the 3 double-spaced repetition strategies. Cloned questions emphasize the same learning points as the original (base) question and are designed to measure the transfer of knowledge from one clinical situation to another. In quarter 8 of the study, physicians in all groups will receive questions cloned from their selected baseline questions. We will then examine the effects of different numbers and intervals of spaced repetition on transfer of knowledge from one clinical scenario to another. We will present data on participant retention rates through the first 6 study quarters, and preliminary analysis of learning in the different groups. To date, spaced repetition has not been systematically evaluated in large cohorts of practicing physicians. This study can inform the development and utilization of spaced repetition strategies by health professions educators across the continuum. The results of this study will help ABFM determine a strategy for and the potential added value of spaced repetition in the lifelong learning and self-assessment component of continuous certification. The authors wish to thank Zachary Morgan, Emily Banik, Matt Wilhoite, and Prasad Chodavarapu for their assistance with data acquisition and analysis.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.016 | 0.017 |
| Meta-epidemiology (narrow) | 0.002 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.002 |
| Bibliometrics | 0.002 | 0.002 |
| Science and technology studies | 0.002 | 0.001 |
| Scholarly communication | 0.001 | 0.001 |
| Open science | 0.002 | 0.002 |
| Research integrity | 0.001 | 0.001 |
| Insufficient payload (model declined to judge) | 0.003 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".