Automated de-identification and unstructured textual electronic medical record data in Manitoba
Bibliographic record
Abstract
Introduction: Unstructured textual electronic medical record (EMR) data contain valuable patient details that can benefit health research. Personal health information (PHI) must be de-identified for EMR data to be used for secondary purposes. A considerable amount of de-identification research has been conducted using existing synthetic, de-identified, and annotated data sets. To date, little is known about how existing de-identification literature applies to unstructured EMR data in Manitoba. Objectives: The research objectives were to: 1) categorize the types and frequency of PHI in Manitoba EMR data, 2) assess the applicability of de-identification literature on Manitoba EMR data, and 3) test how NLM-Scrubber, an existing de-identification tool validated to be successful, redacts PHI in Manitoba EMR data. Methods: The Manitoba data set comprised of 750 unstructured textual EMR encounter notes from 2003 to 2017 from the Manitoba Primary Care Research Network. In-scope PHI included name, personal health information number, address, phone number, and date (excluding year). Two annotators tagged PHI in the Manitoba data using the Visual Tagging Tool. Comparison of Manitoba data and the 2014 i2b2 corpus examined note compilation and PHI prevalence. NLM-Scrubber’s de-identification of Manitoba data was assessed using performance measures and tested against the null hypothesis that NLM-Scrubber will recall ≥87% of PHI in Manitoba data. Results: The Manitoba EMR data contained 3,314 PHI instances, demonstrating 1.6% PHI prevalence. All in-scope PHI types were present. The Manitoba data offered more independent notes and broader variety of note types than the i2b2 corpus. The Manitoba EMR data contained nearly twice as many name PHI instances as the i2b2 corpus (62% and 32%, respectively) but fewer date instances (31% and 55%, respectively). NLM-Scrubber’s PHI recall was 75.4% (95% CI, 72.9-77.8%), leading to rejection of the null hypothesis. Conclusion: Direct and indirect PHI represent a small proportion of Manitoba EMR data. De-identification literature may have limited applicability to Manitoba EMR data. NLM-Scrubber may not be acceptable for use on Manitoba EMR data due to its low recall performance. Attention should be directed to trained machine learning solutions that enable customization, adjustment of rule-based methods, and pseudo PHI to protect patient privacy.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.004 | 0.025 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.006 | 0.008 |
| Science and technology studies | 0.002 | 0.001 |
| Scholarly communication | 0.002 | 0.001 |
| Open science | 0.001 | 0.004 |
| Research integrity | 0.001 | 0.001 |
| Insufficient payload (model declined to judge) | 0.002 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".