Bibliographic record
Abstract
The restricted availability of free-text clinical notes and embedding-models trained on clinical notes is a bottleneck in deploying machine learning in clinical settings. To ameliorate concerns regarding the confidentiality of patient data, researchers are developing methods which automatically remove sensitive personal information from notes. While these methods appear to perform exceedingly well, often with reported precision and recall well above 95%, automated approaches for de-identification are still not trusted by clinicians because there remains non-zero risk. In this work, we present fundamental limitations associated with current approaches to de-identification that cannot be solved by incremental improvements to model performance. These limitations stem from the fact that current approaches are all trained in a supervised manner. To address these limitations, this thesis proposes the first unsupervised approach to the de-identification on free-text clinical notes. The proposed algorithm replaces all tokens with other tokens pseudo-randomly sampled from trained embeddings and is most useful for tasks where humans are not required to read the de-identified notes (e.g., training word embeddings for public release, piloting the feasibility of end-to-end machine learning models). Our approach successfully side-steps the issues facing supervised approaches (e.g., having to decide what constitutes sensitive personal information). The second part of the thesis argues for an expansion to the scope of clinical de-identification. Whereas existing de-identification approaches focus solely on protecting patients’ identities, we argue that de-identification should also focus on protecting the identity of healthcare providers. First, we demonstrate that authorship attribution in clinical notes is a very easy task when compared to many traditional author attribution datasets. Despite the need for specialized and improved author obfuscation techniques, the data to develop such techniques is difficult to obtain due to privacy concerns; it is impractical to manually label paired sentences and difficult to crowd-source the task given data-sharing limitations. To enable the development of automated means of evaluating semantic relatedness, we developed a novel sentence-pair dataset ordered by semantic relatedness. This dataset can serve as a catalyst for future author obfuscation evaluation, and we draw insights from this dataset to better understand existing work.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.010 | 0.001 |
| Meta-epidemiology (narrow) | 0.000 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.000 |
| Bibliometrics | 0.000 | 0.001 |
| Science and technology studies | 0.002 | 0.000 |
| Scholarly communication | 0.001 | 0.000 |
| Open science | 0.002 | 0.000 |
| Research integrity | 0.001 | 0.003 |
| Insufficient payload (model declined to judge) | 0.003 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; both teacher heads agree on what is shown here.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".