Guidelines for Creating Written Clinical Reasoning Exams: Insight from a Delphi Study
Bibliographic record
Abstract
Clinical reasoning is an essential skill to be learned by medical students, and thus requires to be assessed. Although written exams are widely used as one of the tools to assess clinical reasoning, there are no specific guidelines to help an exam writer to develop good clinical reasoning assessment questions. Therefore, we conducted a modified Delphi study to identify guidelines for writing questions that assess clinical reasoning. Participants were identified from: 1) the literature on clinical reasoning (i.e., people who wrote about clinical reasoning and assessment), 2) the people responsible for assessment in Canadian medical faculties, and 3) a snowball sampling strategy. Thirty-two question-writing guidelines were drawn from the literature and adapted by the team members. Participants were asked to indicate on a ten-point Likert scale their perceived importance of each guideline, and, starting in the second round, the relevance of each guideline in five assessment contexts. A total of three rounds were conducted. Response rates were 24%, 57%, and 62% for each round, respectively. Consensus about the importance of the guidelines (interquartile range < 2.5) was reached for all but four guidelines. Four guidelines were identified as important (median ≥ 9 on ten-point scale): the question should be based on a clinical case, the question represents a challenge achievable for the student, the correction scale (i.e., scoring grid) is explicit, and a panel of experts revises the questions. A large number of guidelines seem relevant for written-exam clinical reasoning assessment questions. We are considering grouping those guidelines into categories to create a simple tool for use by medical educators in the design of written-exam clinical reasoning assessment questions. The next step will then be to collect evidence of validity about this tool: Does it really help to build questions that assess clinical reasoning?
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.172 | 0.235 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.004 | 0.003 |
| Science and technology studies | 0.006 | 0.006 |
| Scholarly communication | 0.007 | 0.008 |
| Open science | 0.003 | 0.010 |
| Research integrity | 0.003 | 0.004 |
| Insufficient payload (model declined to judge) | 0.003 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".