The impact of cueing on written examinations of clinical decision making: a case study
Bibliographic record
Abstract
CONTEXT: Selected-response (SR) formats (e.g. multiple-choice questions) and constructed-response (CR) formats (e.g. short-answer questions) are commonly used to test the knowledge of examinees. Scores on SR formats are typically higher than scores on CR formats. This difference is often attributed to examinees being cued by options within an SR question, but there could be alternative explanations. The purpose of this study was to expand on previous work with regards to the cueing effect of SR formats by directly contrasting conditions that support cueing versus memory of previously seen questions. METHODS: During an objective structured clinical examination, students (n = 144) completed two consecutive stations in which they were presented with the same written cases but in different formats. Group 1 students were presented with CR questions followed by SR questions. Group 2 students were presented with questions in reverse order. Participants were asked to describe their testing experience. RESULTS: Selected-response scores (M = 4.21/10) were statistically higher than the CR scores (M = 3.82/10). However, there was no significant interaction between sequence and format (F(1,142) = 1.59, p = 0.21, ηp2 = 0.01) with scores increasing from 3.49/10 to 4.06/10 in the group that started with CR and decreasing (4.38/10-4.15/10) in the group that started with SR first. Correlations between SR scores and CR scores were high (CR first = 0.78, SR first = 0.89). Questionnaire results indicated that students felt the SR format was easier and led to cueing. CONCLUSION: To better understand test performance, it is important to know how different response formats could influence results. Because SR scores were higher than CR scores, irrespective of the format seen first, the pattern is consistent with what would be expected if cueing rather than memory for prior questions led to higher SR scores. This could have implications for test designers, especially when selecting question formats.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.002 | 0.043 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.000 | 0.000 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".