Predicting Performance on the Medical Council of Canada Qualifying Exam Part II
Bibliographic record
Abstract
BACKGROUND: Being able to predict which residents will likely be unsuccessful on high-stakes exams would allow residency programs to provide early intervention. PURPOSE: To determine whether measures of clinical performance in clerkship (in-training evaluation reports) and first year of residency (program director ratings) predict pass-fail performance on the Medical Council of Canada Qualifying Exam Part II (MCCQE Part II). METHODS: Residency program directors assessed the performance of our medical school graduates (Classes 2004-2007) at the end of the 1st postgraduate year. We subsequently collected clerkship in-training evaluation reports for these graduates. Using a neutral third party and unique codes, an anonymous dataset containing clerkship, residency, and MCCQE Part II performance scores was created for our use. Data were analyzed using descriptive statistics, correlations, receiver operating characteristics, and the Youdin index. Regression was also performed to further study the relationship among the variables. RESULTS: Complete data were available for 78.6% of the graduates. Of these participants, 94% passed the licensing exam on their first attempt. Receiver operating characteristics revealed that the area under the curve for clerkship in-training evaluation reports was 0.67 (p<.05) and 0.66 (p<.05) for residency program directors assessments. Corresponding Youdin indices for in-training evaluation reports and residency program director assessments were 0.30 and 0.23, respectively. CONCLUSIONS: Although clerkship in-training evaluation reports and residency program director ratings are significant predictors of pass-fail performance on the MCCQE Part II, the effectiveness of each one to predict pass-fail performance was relatively small. Reasons for these findings are discussed.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.010 | 0.036 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.000 | 0.000 |
| Science and technology studies | 0.001 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.003 |
| Insufficient payload (model declined to judge) | 0.001 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".