Mixed methods analysis of an automated email audit and feedback intervention for fostering (emergency) physician reflection
Bibliographic record
Abstract
Background physician refelection requires personalized, timely and growth-oriented feedback. Iterative learning from multiple low-pressure events can be personalized to target areas of weakness and show sequential growth. Since emergency physicians typically work individually to deliver episodic care, opportunities for them to obtain iterative feedback on their clinical performace is often limited. Our study sought to evaluate whether physician reflection is facilitated through the 72hr re-admission alert received by emergency physicians in the Calgary zone. Implementation The 72-hr readmission alert is already part of feedback received in the Calgary Zone. Our study was specifically looking at understanding the utility of these alerts to emergency physicians through qualitative interviews. Our team of two interviewers (DA and CP) collected and banked the data through anonymized one-on-one interviews. Themes from these interviews will be used to guide future adjustments made to the alert and dictate it’s future role in emergency physician feedback. Current changes based on preliminary data have included the ability to customize re-admission alert time-frames based on personal preference. We are currently in the process of analyzing the themes that will shape further improvements made to the alert. Evaluation Methods This mixed methods realist evaluation consisted of two sequential phases: an initial quantitative phase examining the general features of 72-hr readmission alerts sent over a 1-year period (4024 alerts from May 2017-2018) and a subsequent qualitative phase involving 17 semi-structured interviews to generate “context-mechanism-outcome” (CMO) statements to guide refinement of our program theory. Results CMO statements revealed emergency physician stakeholders were concerned that the alert impacted personnel decisions, changed patient return expectations and didn’t involve consulting services. Physicians, who didn’t believe alerts were involved in personnel decisions, were more likely to pursue balanced reflection/acquisition after each alert when receiving illness related returns. Conversely, physicians, who believed alerts were involved in performance assessment/hiring decisions, were more likely to defensively change their practice. Commonly cited areas of improvement were the ability to personally adjust time criteria for alerts and involving consulting services in feedback. Advice and Lessons Learned It is essential to partner with local departments who can use formal (newsletters) and informal (word of mouth) avenues to encourage participation in the study. Participant anonymity must be emphasized when recruiting for qualitative interviews in order to receive the full scope of perspectives. Clear and concise scripts highlighting the objective of each question can ensure the quality of responses received and help interviewers probe further into the topic when necessary. When performing quality improvement studies on formal feedback mechanisms, faculty leadership buy-in is essential in order to ensure a safe environment for all participants.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.073 | 0.128 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.003 | 0.003 |
| Science and technology studies | 0.003 | 0.002 |
| Scholarly communication | 0.003 | 0.002 |
| Open science | 0.002 | 0.002 |
| Research integrity | 0.001 | 0.002 |
| Insufficient payload (model declined to judge) | 0.005 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".