A Comparison of Remote vs In-Person Proctored In-Training Examination Administration for Internal Medicine
Bibliographic record
Abstract
PURPOSE: In response to COVID-19, the American College of Physicians provided residents the option to complete the 2020 Internal Medicine In-Training Examination (IM-ITE) via in-person and remote proctoring. This study evaluated the extent to which scores obtained from both testing modalities were comparable. METHOD: Data were analyzed from residents from all U.S.-based Accreditation Council for Graduate Medical Education-accredited IM residency programs and participating Canadian and international programs who completed the IM-ITE in 2020. The final sample contained 27,115 IM residents: 9,205 postgraduate year (PGY) 1, 9,332 PGY-2, and 8,578 PGY-3. Testing modality, gender, PGY, time spent on assessment, and native language were used to predict percent-correct scores in a multilevel regression model. This model included all main effects and all 2-way interactions between testing modality and each resident-level demographic variable, allowing those effects to be controlled for. RESULTS: Of 27,115 residents studied, 11,354 (42%) tested remotely and 15,761 (58%) in person. Across the parameters of interest (main effect of testing modality and 2-way interactions), the only statistically significant effects were the interaction effects between testing mode (interaction effect: -0.61; 95% confidence interval [CI], -1.01 to -0.21) and PGY (interaction effect: -0.54; 95% CI, -0.95 to -0.13) ( P = .002). Differences between in-person and remote predicted scores were slightly larger for PGY-1 than for PGY-2 and PGY-3 residents, but the magnitude of these differences across residency training was well under one percentage point. Because these statistically significant effects were deemed educationally nonsignificant, the study concluded that performance did not substantively differ across in-person and remote examinees. CONCLUSIONS: Residents taking the 2020 IM-ITE performed similarly across in-person and remote proctoring. This study provides evidence of score comparability across the 2 testing modalities and supports continued use of remote proctoring for the IM-ITE.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.006 | 0.052 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.001 | 0.001 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.001 | 0.001 |
| Open science | 0.001 | 0.001 |
| Research integrity | 0.000 | 0.001 |
| Insufficient payload (model declined to judge) | 0.003 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".