Generative <scp>AI</scp> in Otolaryngology Residency Personal Statement Writing: A Mixed‐Methods Analysis
Bibliographic record
Abstract
OBJECTIVE: Generative Artificial Intelligence (GAI) interfaces have rapidly integrated into various societal domains. Widespread accessibility of GAI for drafting personal statements poses challenges for evaluators to gauge writing ability and personal insight. This study aims to compare the quality of GAI-generated personal statements to those written by successful applicants in OHNS residency programs, via integration of statistical and qualitative thematic analyses. METHODS: Personal statements were collected from successful OHNS residency applicants. Characteristic extraction from submitted statements was used to generate GAI-written personal statements using ChatGPT 4.0. All statements were blindly reviewed by 21 experienced evaluators on a 10-point Likert scale of authenticity, readability, personability, and overall quality. Thematic analysis of qualitative reviewer comments was conducted to extract deeper insights into evaluators' perceptions. Quantitative results were compared using independent t-tests, while thematic coding was performed inductively using NVivo software. RESULTS: GAI-generated personal statements significantly outperformed applicant-written statements in all assessed domains, including authenticity (7.67 vs. 7.05, p = 0.002), readability (8.03 vs. 7.49, p = 0.002), personability (7.33 vs. 6.72, p = 0.004), and overall score (7.49 vs. 6.90, p = 0.005). Thematic analysis revealed that GAI statements were seen as "well-constructed but generic," while applicant statements were often "verbose and lacked focus." Additionally, reviewers noted concerns regarding personal insight and engagement in AI-generated statements. CONCLUSION: GAI-generated personal statements were rated more favorably across all domains, raising critical questions about the future of personal statements in the residency application process. While AI in medical education continues to evolve, clear guidelines on its ethical use in residency applications are essential. LEVEL OF EVIDENCE: N/A.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.002 | 0.001 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.001 | 0.002 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.001 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".