Using Primary Care Text Data and Natural Language Processing to Monitor COVID-19 in Toronto, Canada
Bibliographic record
Abstract
A bstract Objective To investigate whether a rule-based natural language processing (NLP) system, applied to primary care clinical text data, can be used to monitor COVID-19 viral activity in Toronto, Canada. Design We employ a retrospective cohort design. We include primary care patients with a clinical encounter between January 1, 2020 and December 31, 2020 at one of 44 participating clinical sites. Setting and Context The study setting is Toronto, Canada. During the study timeframe the city experienced a first wave of COVID-19 in spring 2020; followed by a second viral resurgence beginning in the fall of 2020. Methods and Data Study objectives are descriptive. We use an expert derived dictionary, pattern matching tools and a contextual analyzer to classify documents as 1) COVID-19 positive, 2) COVID-19 negative, or 3) unknown COVID-19 status. We apply the COVID-19 biosurveillance system across three primary care electronic medical record text streams: 1) lab text, 2) health condition diagnosis text and 3) clinical notes. We enumerate COVID-19 entities in the clinical text and estimate the proportion of patients with a positive COVID-19 record. We construct a primary care COVID-19 NLP-derived time series and investigate its correlation with other external public health series: 1) lab confirmed COVID-19 cases, 2) COVID-19 hospitalizations, 3) COVID-19 ICU admissions, and 4) COVID-19 intubations. Results Over the study timeframe 1,976 COVID-19 positive documents, and 277 unique COVID-19 entities were identified in the lab text. 539 COVID-19 positive documents and 121 unique COVID-19 entities were identified in the health condition diagnosis text. And 4,018 COVID-19 positive documents, and 644 unique COVID-19 entities were identified in the clinical notes. A total of 196,440 unique patients were observed over the study timeframe, of which 4,580 (2.3%) had at least one positive COVID-19 document in their primary care electronic medical record. We constructed an NLP-derived COVID-19 time series describing the temporal dynamics of COVID-19 positivity status over the study timeframe. The NLP derived series correlates strongly with external public health series under investigation. Conclusions Using a rule-based NLP system we identified hundreds of unique COVID-19 entities, and thousands of COVID-19 positive documents, across millions of clinical text documents. Future work should continue to investigate how high quality, low-cost, passively collected primary care electronic medical record clinical text data can be used for COVID-19 monitoring and surveillance.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.001 | 0.011 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.003 | 0.006 |
| Science and technology studies | 0.001 | 0.001 |
| Scholarly communication | 0.002 | 0.000 |
| Open science | 0.001 | 0.001 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.002 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".