Situational Judgement Testing (SJT) in Licentiate Assessment in Canada
Bibliographic record
Abstract
The Medical Council of Canada (MCC) is reviewing medical licensing, asking whether there are gaps in short-term and long-term processes, and how these gaps might be addressed. As a team, we were asked to reflect on the current state of science regarding Situational Judgement Testing (SJT) and to consider what role this form of testing could have in high stakes licensing activity in Canada. To that end, this white paper is not a systematic review but a critical reflection on the strengths and weaknesses of SJTs and their potential role in licentiate assessment both today and tomorrow. Simply put, situational judgement testing is about questions that assess a candidate’s ability to apply their knowledge and skill to solve problems in the domain in which they are being tested. SJTs therefore expect a candidate to be able to: recognize what the problem is in the situation presented to them; identify possible actions and rank them according to suitability; and select a course of action from the possibilities suggested by their interpretation of the situation. They don’t ask what the best way to do something in general is, rather they ask ‘how would you act in this particular situation?’ SJTs are complex assessment tools – a wrong answer does not map to a single gap in knowledge or skill but to a failure somewhere in the interactions of their interpretation, knowledge, and decision-making in the situation presented to them. Thus, while SJTs are closer to practice (for instance according to Miller’s pyramid)1 than knowledge testing instruments such as MCQs (and can therefore be argued to have good inferential validity), quite what they are assessing other than the aggregate and vague construct of judgment is often left unspecified. In order to answer our question of what role SJTs might play in licentiate assessment, we explore: 1) what they can and cannot do, 2) what their affordances can offer licentiate assessment, and 3) how practically they might be employed in licentiate assessment. While the main thrust of this paper is about using SJTs in LMCC exams, we also hope that MCC will consider some of the recommendations that may pertain to various forms of ongoing professional assessment and revalidation.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.103 | 0.315 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.002 | 0.002 |
| Bibliometrics | 0.008 | 0.015 |
| Science and technology studies | 0.008 | 0.012 |
| Scholarly communication | 0.011 | 0.007 |
| Open science | 0.006 | 0.012 |
| Research integrity | 0.003 | 0.006 |
| Insufficient payload (model declined to judge) | 0.006 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".