Using Lbl2Vec and BERTopic for Semi-Supervised Detec-tion of Professionalism Aspects in a Constructed-Response Situational Judgment Test
Bibliographic record
Abstract
Most medical schools in the United States and Canada evaluate applicants based on both academic achievements and non-cognitive (soft) skills such as professionalism. Commonly, medical school applicants are required to complete Casper—an online constructed-response situational judgment test developed and administered by Altus Assessments to assess various aspects of professionalism such as ethics, empathy, resilience, and collaboration. Examinees writing Casper are given a set of real-life, hypothetical scenarios and asked to explain how they would handle the situation and their reasoning behind it. Since each Casper scenario focuses on multiple aspects of professionalism, examinees do not have to target any specific aspect in their responses. To ensure the correct use and interpretation of Casper scores, the Altus team needs to reliably evaluate whether examinees’ responses for each scenario address the anticipated aspects of professionalism identified by subject mat-ter experts. In this study, we demonstrate how natural language processing (NLP) methods can be employed to characterize the meaning and context of written responses in relation to different aspects of professionalism. Using the Lbl2Vec and BERTopic algorithms, we performed a semi-supervised analysis of written responses from a large sample of medical school applicants (n = 635,106) by clustering the responses into latent topics based on a set of predefined keywords and labels for each professionalism aspect. Our analysis yielded substantive topics that were mostly aligned with the anticipated professionalism aspects. Furthermore, in terms of topical prevalence, some aspects of professionalism (e.g., self-awareness) appeared to be more dominant than the other aspects in the responses. Overall, our proposed approach delivers a feasible mechanism to identify overlapping topics and aspects to maintain test quality and improve human rater training and monitor-ing in constructed-response situational judgment tests.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.001 | 0.005 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.000 | 0.000 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.008 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".