Assessing and improving the accuracy of surveillance case definitions using administrative data

BACKGROUND Keeping pace with the rapidly evolving demands of infectious disease monitoring requires constant advances in surveillance methodology and infrastructure. A promising new method is syndromic surveillance, where health department staff, assisted by automated data acquisition and statistical alerts, monitor health indicators in near real-time. Several syndromic surveillance systems use diagnoses in administrative databases. However, physician claim diagnoses are not audited, and the effect of diagnostic coding variation on surveillance case definitions is not known. Furthermore, syndromic surveillance systems are limited by high false-positive (FP) rates. Almost no effort has been made to reduce FP rates by improving the positive predictive value (PPV) of surveilled data. OBJECTIVES 1) To evaluate the feasibility of identifying syndrome cases using diagnoses in physician claims. 2) To assess the accuracy of syndrome definitions based on diagnoses in physician claims. 3) To identify physician, patient, encounter and billing characteristics associated with the PPV of syndrome definitions. METHODS & RESULTS STUDY 1: We focused on a subset of diagnoses from a single syndrome (respiratory). We compared cases and non-cases identified from physician claims to medical charts. A convenience sample of 9 Montreal-area family physicians participated. 3,526 visits among 729 patients were abstracted from medical charts and linked to physician claims. The sensitivity and PPV of physician claims for identifying respiratory infections were 0.49, 95%CI (0.45, 0.53) and 0.93, 95%CI (0.91, 0.94). This pilot work demonstrated the feasibility of the proposed method and contributed to planning a full-scale validation of several syndrome definitions. STUDY 2: We focused on 5 syndromes: fever, gastrointestinal, neurological, rash, and respiratory. We selected a random sample of 3,600 physicians practicing in the province of Quebec in 2005-2007, then a stratified random sample of 10 visits per physician from their claims. We obtained chart diagnoses for all sampled visits through double-blinded chart reviews. Sensitivity, specificity, PPV, and negative predictive value (NPV) of syndrome definitions based on diagnoses in physician claims were estimated by comparison to chart review. 1,098 (30.5%) physicians completed the chart review and 10,529 visits were validated. The sensitivity of syndrome definitions ranged from 0.11, 95%CI (0.10, 0.13) for fever to 0.44, 95%CI (0.41, 0.47) for respiratory syndrome. The specificity and NPV were high for all syndromes. The PPV ranged from 0.59, 95%CI (0.55, 0.64) for fever to 0.85, 95%CI (0.83, 0.88) for respiratory syndrome. STUDY 3: We focused on the 4,330 syndrome cases identified from the claims of the 1,098 physicians who participated in study 2. We estimated the association between claim-chart agreement and physician, patient, encounter and billing characteristics using multivariate logistic regression. The likelihood of the medical chart agreeing with the physician claim about the presence of a syndrome was higher when the physician had billed many visits for the same syndrome recently (RR per 10 visits, 1.05; 95%CI, 1.01-1.08), had a lower workload (RR per 10 claims, 0.93; 95%CI, 0.90-0.97), and when the patient was younger (RR per 5 years, 0.96; 95%CI, 0.94-0.97) and less socially deprived (RR most vs least deprived, 0.76; 95%CI, 0.60-0.95). CONCLUSIONS This was the first population-based validation of syndromic surveillance case definitions based on diagnoses in physician claims. We found that the sensitivity of syndrome definitions was low, the PPV was moderate to high, and the specificity and NPV were high. We identified several physician, patient, encounter and billing characteristics associated with the PPV of syndrome definitions, many of which are readily accessible to public health departments and could be used to reduce the FP rate of syndromic surveillance systems.

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.