Risk Heterogeneity and the Illusion of Waning Vaccine Efficacy

Ideas and OpinionsMarch 2022Risk Heterogeneity and the Illusion of Waning Vaccine EfficacyFREECorrection(s) for this article:CorrectionsFeb 2023Correction: Risk Heterogeneity and the Illusion of Waning Vaccine EfficacyFREEMichael P. Fay, PhD, Sally Hunsberger, PhD, and Dean Follmann, PhDMichael P. Fay, PhDNational Institute of Allergy and Infectious Diseases, Bethesda, Maryland (M.P.F., S.H., D.F.)., Sally Hunsberger, PhDNational Institute of Allergy and Infectious Diseases, Bethesda, Maryland (M.P.F., S.H., D.F.)., and Dean Follmann, PhDNational Institute of Allergy and Infectious Diseases, Bethesda, Maryland (M.P.F., S.H., D.F.).Author, Article, and Disclosure Informationhttps://doi.org/10.7326/M21-3609 SectionsAboutVisual AbstractPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissions ShareFacebookTwitterLinkedInRedditEmail Critical scientific questions about COVID-19 vaccines are whether the vaccine effect will wane over time, whether they will work on new variants, and whether booster shots are needed. As reports of the durability of vaccine protection become available (1–4), general scientific readers should be aware of a subtle issue, known in the statistics literature as frailty effects or in the vaccine literature as depletion-of-susceptibles bias (5–8). The issue is that a study with higher- and lower-risk participants (in other words, risk heterogeneity) can create an illusion of waning vaccine efficacy (VE) over time even when the effect of the vaccine on each individual is not changing, or it can exaggerate the estimated magnitude of true individual waning VE.Vaccine efficacy is defined as the percentage reduction in disease incidence among a vaccinated group compared with an unvaccinated group over a specified time period. It is typically measured in 2 previously uninfected populations: one newly vaccinated and the other not (the control population). Randomization at baseline is used to balance the risk factors for future infection. As an example, if 10% of a control group and 7% of a vaccinated group get the disease over 10 months of follow-up, the VE over the 10-month period is 30% (1 − 7%/10%). A complexity arises when we want to evaluate how VE changes over time, such as comparing efficacy for the first 5 months compared with the second 5 months.When measuring VE at different time periods since vaccination, we typically exclude participants who have gotten the disease in earlier time periods in our calculations, because we want to estimate the vaccine's effect on people who have not yet experienced the disease within the study. A complicating issue is that people differ in immune functionality (some individuals are more frail and have a less robust immune system) and exposure risk (for example, some individuals may have high risk, such as health care workers or nursing home residents). Randomization creates 2 populations (vaccine and control) with the same proportions of frail and/or high-risk individuals at the start of the studies but, if the vaccine works, in the later periods there will be lower proportions of frail/high-risk individuals left in the control group who did not have the disease in early periods.Consider a simple example in the Figure, based on a model with 2 types of individuals: low-risk individuals and high-risk individuals—where the risk for disease is 4 times greater for the high-risk group. In the example, an individual's VE is about 75% regardless of risk group or period. At randomization, the control and vaccine groups have equal proportions of high-risk participants, so the early population VE estimate is 75%, and it is just the average of individual VEs. But in the late period, many fewer high-risk participants are in the control group, because they did not make it disease-free to the late period. Unlike the start of the study, we are now comparing "apples," or higher-risk vaccine recipients, to "oranges," or lower-risk control participants. Thus, a smaller proportion of control participants get the disease in the later period. The late period population VE estimate is 62%. The individual VE has not changed over time, and the only difference between the early and late periods is the increased proportion of low-risk people in the control group. This can happen in big studies, too; if each figure in the graph represented 1000 people, the differences in population VE from early to late periods remain. The problem should not markedly affect a study with a low percentage of events (for example, Polack and colleagues [9] had an event rate of <0.5% for the initial VE estimates) because few people get the disease in the early period of the study. For large real-world observational studies, the problem may be more severe because the risk groups are not balanced between unvaccinated and vaccinated participants.Figure. Example of heterogeneity of risk affecting population VE, whereas an individual's VE remains constant across time.For each participant, the individual VE is 75%, and the frail/high-risk participants (orange) are 4 times more likely to get disease than the healthy/low-risk participants (black). Red slashes denote cases of disease that occurred during that period. In the early period in the vaccine group, 1 of 20 in the low-risk group are cases and 4 of 20 in the high-risk group are cases, to give an overall case rate of 5/40, whereas in the control group, 4 of 20 in the low-risk group are cases and 16 of 20 in the high-risk group are cases, to give an overall case rate of 20/40. The early period population VE is thus 1 minus the ratio of rates: VE = 1 – (5/40)/(20/40) = 75%. But in the late period, many fewer frail/high-risk participants are in the control group. The analogous calculation gives an estimated population VE of 1 – (4/35)/(6/20) = 62%. Although the individual VE is always 75% throughout follow-up, the population VEs over time give the illusion that protection is waning. VE = vaccine efficacy. Download figure Download PowerPoint A solution for the heterogeneity problem is to account for the variability of risk in the modeling (see, for example, Hernán and Robins [10]). For the simple example, we could calculate separate VEs for the low-risk populations and the high-risk populations. But accurately identifying risk can be hard. The point of this note is that the larger the proportion of the study that gets the disease, the more important it is to get the risk groups correct. In the excellent study by Pouwels and colleagues (2), they corrected for many potential risk factors (see their Table S1); however, a large proportion of those that started out acting as controls (unvaccinated with no prior infection) got the disease as the study progressed (see their Figure S2). For such substantial disease rates, it is very important that all of the major risk factors are addressed—a tough challenge for real-world studies. Other types of studies to estimate durability of vaccine protection that account for this problem are unethical or impractical (for example, a challenge study where the study population is kept protected from exposure for a long period of time before challenge). One possibility is to have periods where the study population is naturally unexposed and vaccination is spread out over calendar time, as can be done with seasonal influenza (8). Or different studies (or sites) of a vaccine may serendipitously launch into waning and waxing outbreaks. If durability wanes similarly over all settings, it is likely real. Another important clue is whether waning VE tends to track with antibody levels over time. Such tracking suggests a mechanism for waning VE. Finally, studies of different vaccines but with similar populations and attack rates over time should have similar waning, if due to risk heterogeneity. If one vaccine seems durable and the other not, the waning (and durability) is likely real. Although large randomized and observational studies should give the best information about the durability of vaccine protection, they require scrutiny. Apparent waning efficacy of a vaccine in a study may be real or overstated and the potential for illusory waning should be critically assessed and contextualized using other sources of information.References1. Sanderson K. COVID vaccines protect against Delta, but their effectiveness wanes. Nature. 2021. [PMID: 34413527] doi:10.1038/d41586-021-02261-8 CrossrefMedlineGoogle Scholar2. Pouwels KB, Pritchard E, Matthews P, et al; COVID-19 Infection Survey Team. Impact of Delta on viral burden and vaccine effectiveness against new SARS-CoV-2 infections in the UK. medRxiv. Preprint posted online 24 August 2021. doi:10.1101/2021.08.18.21262237 Google Scholar3. Mizrahi B, Lotan R, Kalkstein N, et al. Correlation of SARS-CoV-2 breakthrough infections to time-from-vaccine; preliminary study. medRxiv. Preprint posted online 31 July 2021. doi:10.1101/2021.07.29.21261317 Google Scholar4. Israel A, Merzon E, Schäffer AA, et al. Elapsed time since BNT162b2 vaccine and risk of SARSCoV-2 infection in a large cohort. medRxiv. Preprint posted online 5 August 2021. doi:10.1101/2021.08.03.21261496 Google Scholar5. Vaupel JW, Yashin AI. Heterogeneity's ruses: some surprising effects of selection on population dynamics. Am Stat. 1985;39:176-85. [PMID: 12267300] MedlineGoogle Scholar6. Aalen OO. Effects of frailty in survival analysis. Stat Methods Med Res. 1994;3:227-43. [PMID: 7820293] CrossrefMedlineGoogle Scholar7. Hernán MA. The hazards of hazard ratios. Epidemiology. 2010;21:13-5. [PMID: 20010207] doi:10.1097/EDE.0b013e3181c1ea43 CrossrefMedlineGoogle Scholar8. Lipsitch M, Goldstein E, Ray GT, et al. Depletion-of-susceptibles bias in influenza vaccine waning studies: how to ensure robust results. Epidemiol Infect. 2019;147:e306. [PMID: 31774051] doi:10.1017/S0950268819001961 CrossrefMedlineGoogle Scholar9. Polack FP, Thomas SJ, Kitchin N, et al; C4591001 Clinical Trial Group. Safety and efficacy of the BNT162b2 mRNA covid-19 vaccine. N Engl J Med. 2020;383:2603-2615. [PMID: 33301246] doi:10.1056/NEJMoa2034577 CrossrefMedlineGoogle Scholar10. Hernán MA, Robins JM. Causal Inference: What If. Boca Raton: Chapman & Hall/CRC; 2020. Google Scholar Comments0 CommentsSign In to Submit A Comment Carlos Hernandez-SyuarezUniversidad Francisco Gavidia16 November 2022 Waning VE is not measured as you explain Waning VE is not measured as you explain in your detailed example. In your example, one has a sample with vaccinated and other as a control, then measure VE as the relative risk. That is OK, so far. Then, in your example, infection occurs in time, individuals are removed from the analysis and the VE is measured again, using the remaining sample. That part is wrong. To measure VE the second time, you use the cumulative numbers. Using your example, the second time you measure VE there has been 9 infections accumulated in the vaccinated group, and 26 in the unvaccinated group, giving a 65% VE (not 62%), down from the initial 75%. Randomization takes care from differential exposure (1). But measuring waning VE can be done alternatively, using survival models, not neccesarily requiring "snapshots" at different intervals (2). References: 1. Hernández‐Suárez, Carlos M. "A note on the distribution of the number of vaccinated infected under non‐random mixing conditions." Statistics in medicine 20, no. 13 (2001): 1983-1986. 2. Hernandez-Suarez, Carlos, and Efrèn Murillo-Zamora. "Waning immunity to SARS-CoV-2 following vaccination or infection." Frontiers in medicine 9 (2022). Michael P. Fay, Sally Hunsberger, and Dean FollmannNational Institute of Allergy and Infectious Diseases, Bethesda, Maryland29 November 2022 Author Response to Hernández‐Suárez The main points of our piece were that measuring and interpreting waning vaccine efficacy (VE) can be difficult and to explain depletion-of-susceptibles bias for the general reader. In the process, we proposed comparing VE over time, defining VE as 1 minus hazard ratios in each time period. You stated that this measure of VE was wrong, and we should have used "cumulative numbers" to estimate VE. First, others have argued that the best way to measure durability of protection is to define VE in terms of hazard ratios and compare changing hazard ratios over time (1). Second, to defend our VE definition, let us return to our figure, but now let it represent the data generating model of the expected counts from a trial (to avoid any statistical variability), and to simplify it further, consider only the high-risk group. Using those values, we can define the vaccine efficacy for each time period as 1 minus the discrete time hazard ratio. Discrete hazards measure the probability of an event during a time period, given that you had not had the event by the start of that period. Our example is meant to represent a durable vaccine, and the discrete hazard VE is 75% for both the early and late periods (early discrete hazard VE=1-(4/20)/(16/20)=75% and late discrete hazard VE=1-(3/16)/(3/4)=75%). The VE as 1 minus a hazard ratio is an accepted VE definition, but there are many ways to define VE, and defining the VE as 1 minus the cumulative incidence ratio as you suggest is one of several acceptable definitions (2,3). For our example, the cumulative incidence VE for the high-risk individuals is 75% for the early period, and 63.2% through both periods (early cumulative incidence VE=1-(4/20)/(16/20)=75% and overall cumulative incidence VE=1-(7/20)/(19/20)=63.2%). Thus, constant discrete hazard ratios for each period leads to decreasing cumulative incidence VE over time. We could have created an example where the early period cumulative incidence VE equals the overall cumulative incidence VE, but that would require unequal within-period hazard ratios. We feel confident that defining VE using hazard ratios is better for examining vaccine durability than defining VE using cumulative incidence ratios, because hazards define the risk at each point in time, and for studying waning we want to compare risk at different points in time. REFERENCES Lin, D. Y., Zeng, D., Gu, Y., Krause, P. R., & Fleming, T. R. (2022). Reliably Assessing Duration of Protection for Coronavirus Disease 2019 Vaccines. The Journal of Infectious Diseases. https://doi.org/10.1093/infdis/jiac139 Halloran, M. E., Struchiner, C. J., and Longini Jr, I. M. (1997). Study designs for evaluating different efficacy and effectiveness aspects of vaccines. American journal of epidemiology, 146(10), 789-803. Hudgens, M. G., Gilbert, P. B., and Self, S. G. (2004). Endpoints in vaccine trials. Statistical methods in medical research, 13(2), 89-114. Author, Article, and Disclosure InformationAffiliations: National Institute of Allergy and Infectious Diseases, Bethesda, Maryland (M.P.F., S.H., D.F.).Disclosures: Disclosures can be viewed at www.acponline.org/authors/icmje/ConflictOfInterestForms.do?msNum=M21-3609.Corresponding Author: Michael P. Fay, MD, National Institute of Allergy and Infectious Diseases, 5601 Fishers Lane, Room 4C40/MSC 9820, Bethesda, MD 20892; e-mail, [email protected]nih.gov.Author Contributions: Conception and design: M.P. Fay, D. Follmann.Analysis and interpretation of the data: M.P. Fay, D. Follmann, S. Hunsberger.Drafting of the article: M.P. Fay.Critical revision for important intellectual content: M.P. Fay, D. Follmann, S. Hunsberger.Final approval of the article: M.P. Fay, D. Follmann, S. Hunsberger.Statistical expertise: M.P. Fay, D. Follmann, S. Hunsberger.Collection and assembly of data: M.P. Fay.Correction: This article was amended on 27 December 2022 to correct an error in the equation in the second paragraph. A correction has been published (doi:10.7326/L22-0508).This article was published at Annals.org on 21 December 2021. PreviousarticleNextarticle Advertisement FiguresReferencesRelatedDetailsSee AlsoCorrection: Risk Heterogeneity and the Illusion of Waning Vaccine Efficacy Metrics Cited byCorrection: Risk Heterogeneity and the Illusion of Waning Vaccine EfficacyEffectiveness of COVID-19 Vaccines Over Time Prior to Omicron Emergence in Ontario, Canada: Test-Negative Design StudyReduced Odds of Severe Acute Respiratory Syndrome Coronavirus 2 Reinfection After Vaccination Among New York City Adults, July 2021–November 2021 March 2022Volume 175, Issue 3Page: 444-445KeywordsCOVID-19Immune systemImmunityMedical risk factorsObservational studiesRandomized trialsVaccines ePublished: 21 December 2021 Issue Published: March 2022 PDF downloadLoading ...

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.