Using Machine Learning to Measure Specialist Wait Times from Family Physicians’ Electronic Medical Records Linked to Ontario Health Administrative Data
Bibliographic record
Abstract
IntroductionReliable information about the time spent waiting for health care services is a critical metric for measuring health system performance. Wait times are a useful measure of access to various health care sectors. Alongside the increased adoption of electronic medical records (EMR) by Canadian family physicians (FP), is the secondary use of FP EMR data for research. However, using FP EMR data can be challenging in its unstructured, free-text format. Objectives and ApproachOur objective was to identify the target specialist physician type from the EMR FP referral note and then calculate wait times from a FP referral to a specialist physician visit. We used FP EMR data and linked to Ontario, Canada health administrative data (called EMRPC). EMRPC collects the entire clinical record from patients including the content of FP referral notes. We used machine learning (ML) methods to identify the type of specialist physician in which the referral was intended. Labels to test the ML methods were created from physicians’ claims data. Wait times were calculated from the FP EMR referral note date to the specialist physician claim date in administrative data. ResultsOur ML models’ ability to classify 2016 FP EMR referral notes to selected medical and surgical specialists achieved sensitivity and positive predictive values ranging from the high 70s to low 80s.Compared to earlier analyses from 2008, we observed a similar relative ordering to see specific specialist physicians. Overall, the median wait times have increased by 14 days on average, with a maximum increase of 28 days to see a gastroenterologist. Conclusion / ImplicationsThe accuracy of ML on unstructured FP EMR data is high, thereby providing a mechanism to “codifying” information in a timely manner. This information can help inform decision makers and providers about which patients or FP practices are experiencing long wait times in seeing specialist physicians.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.003 | 0.022 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.001 |
| Bibliometrics | 0.003 | 0.004 |
| Science and technology studies | 0.001 | 0.000 |
| Scholarly communication | 0.001 | 0.001 |
| Open science | 0.001 | 0.001 |
| Research integrity | 0.000 | 0.001 |
| Insufficient payload (model declined to judge) | 0.002 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".