Decision letter: Reverse translation of adverse event reports paves the way for de-risking preclinical off-targets

Article Figures and data Abstract eLife digest Introduction Results Discussion Materials and methods References Decision letter Author response Article and author information Metrics Abstract The Food and Drug Administration Adverse Event Reporting System (FAERS) remains the primary source for post-marketing pharmacovigilance. The system is largely un-curated, unstandardized, and lacks a method for linking drugs to the chemical structures of their active ingredients, increasing noise and artefactual trends. To address these problems, we mapped drugs to their ingredients and used natural language processing to classify and correlate drug events. Our analysis exposed key idiosyncrasies in FAERS, for example reports of thalidomide causing a deadly ADR when used against myeloma, a likely result of the disease itself; multiplications of the same report, unjustifiably increasing its importance; correlation of reported ADRs with public events, regulatory announcements, and with publications. Comparing the pharmacological, pharmacokinetic, and clinical ADR profiles of methylphenidate, aripiprazole, and risperidone, and of kinase drugs targeting the VEGF receptor, demonstrates how underlying molecular mechanisms can emerge from ADR co-analysis. The precautions and methods we describe may enable investigators to avoid confounding chemistry-based associations and reporting biases in FAERS, and illustrate how comparative analysis of ADRs can reveal underlying mechanisms. https://doi.org/10.7554/eLife.25818.001 eLife digest New treatments are tested in clinical trials before they are licensed for use in patients, but until the drugs are available for prescribing it's not always possible to identify every side effect. When the drugs enter the clinic, they might be prescribed to patients with multiple medical conditions, or combined with other treatments. The drugs may also be taken for longer periods of time than tested in trials. It is therefore common for new adverse reactions to emerge after a drug is in widespread use. The FDA Adverse Event Reporting System (FAERS) is a surveillance system used in the United States for reporting drug side effects after new treatments have been licensed. Healthcare professionals and patients can submit reports to the database, logging the adverse drug reactions that they have experienced. FAERS currently contains over 8.5 million entries, and is growing all the time. However, Maciejewski et al. show that the database has several shortcomings that are reducing its usefulness. For instance, on average any given drug will have 16 different names in the system; this makes it challenging to group all of the reported side effects so that trends and patterns can be correctly seen. To address this first problem, Maciejewski et al. grouped together drugs according to their active ingredients, rather than their name. This made it much easier to account for subsequent, and more crucial conflating factors such as multiple reports for the same adverse event and patient, or cases where adverse reactions were confused with the diseases that the drugs are trying to treat. For example, diabetes was listed as a side effect for drugs used to treat diabetes. Building on this cleaned-up dataset, Maciejewski et al. monitored how adverse event signals evolve over time and uncovered biases that were hard to see otherwise. For example, side-effects were reported more often when drugs were in the news. More strikingly, this bias affected not only the drug in question, but also other drugs that acted in the same way or on the same molecular target. The computational method developed by Maciejewski et al. allows the data in FAERS to be combined and corrected, making easier to evaluate the safety of different medicines. The link between adverse side effects and the molecular targets of the drug, via the ingredient's chemical structure, furthermore makes it possible to analyze such clinical data reliably by using chemical and genetic information. In the future, this method could also help to identify previously unknown side effects and the biological mechanisms behind them. This could help researchers to develop new drugs with improved side effect profiles. https://doi.org/10.7554/eLife.25818.002 Introduction Safety assessment of drug candidates is crucial for drug discovery, enabling the development of medicines that achieve the desired therapeutic effects with the least risk of adverse side effects. Preclinical regulatory investigations and clinical trials are designed to address safety of drug candidates and eliminate those that do not meet risk-benefit expectations (Cook et al., 2014). However, limited access to large, diverse patient populations in clinical trials, untested drug co-administrations, as often occurs, especially with elderly patients on multiple medications, and development of ADRs associated with chronic treatment, often results in post-marketing labeling and occasional withdrawals (Wysowski and Swartz, 2005; Lasser et al., 2002; Friedman et al., 1999; Downing et al., 2017). Thus, postmarketing pharmacovigilance is essential to track ADRs and ultimately reduce over 1 million serious drug-related side effects that occur each year in the USA. Between 5% and 10% of these ADRs are fatal (Lazarou et al., 1998), and many others cause patient suffering, hospitalization, and increased health system burden (Moore et al., 1998). Indeed, the fatality rate attributed to ADRs puts them among the top causes of death in the USA (over 40,000 in 2011), similar to suicide-related mortality (Hoyert and Xu, 2012). Determinant tools in post-marketing pharmacovigilance are databases that aggregate ADR reports. Foremost among these is the FDA Adverse Event Reporting System (FAERS), which is perhaps the most extensive, and among the most widely accessible of these databases, currently containing over 8.5 million reports and rapidly growing (U.S. Food and Drug Administration, 2016). FAERS and related databases, such as those of the EMEA and of Health Canada, can provide specific ADR phenotypes typical for either individual drug classes or specific indications and can be accessed either directly (U.S. Food and Drug Administration, 2016) or by APIs (RELX Intellectual Properties SA, 2016; U.S. Food and Drug Administration, 2016). These large-scale adverse event databases enable analysis to relate clinical phenotypes and compounds (Tatonetti et al., 2012), and they have been widely used by the clinical community with much impact (O'Connell et al., 2006; Elashoff et al., 2011; Lawrence et al., 2006; Mackey et al., 2007). It is an attractive proposition to exploit the sheer scale of FAERS to detect drug-ADR associations that would otherwise be missed. A challenge in doing so has been the heterogeneous data sources and data conflation in the database. FAERS, while providing a solid frame for reporting, contains redundancies, biases, and conflations that affect its analysis and interpretation (McAdams et al., 2008). Our ability to even correlate drugs with their effects is obscured by something as simple as the tangle of drug synonyms in FAERS - on average 16 different names for medicines containing each active drug ingredient - which can obscure associations. Here, we investigate the effects that these data conflations, inflations, and inaccuracies can have on ADR and mechanistic inference from FAERS, and methods to address them. We begin by mapping drug identifiers in FAERS to normalized chemical structures of their ingredients, which brings together observations over the 'full drug', not just particular drug names and synonyms, which remain incomplete. Mapped to unique chemical structures, we could compute time-resolved profiles of drug-ADR associations, which revealed intriguing comorbidities and similarities of ADRs between drugs, and of their time evolution. We then turned to the origins of and controls for reporting biases in FAERS, considering stimulated reporting and the several different, often non-medical communities that can contribute to FAERS. This was facilitated by a time-evolution analysis of ADR reports, and its correlation with contemporary news events. We illustrate how these biases can ramify with in-depth analysis of FAERS content on two COX-2 inhibitors, rofecoxib and celecoxib, and with two PPAR-γ agonists, rosiglitazone and pioglitazone. As examples of how these analyses can link ADRs to specific targets, we consider the differential ADR profiles of drugs used for the treatment of attention deficit hyperactivity disorder (ADHD), and how their distinct ADRs may be explained partly by molecular targets - a logic that is often used - combined with pharmacokinetic exposure - which is often overlooked. Similarly, we investigate the differentiation of the hypertensive side effects of VEGF-Receptor (VEGF-R2) inhibitors based on their potency and pharmacokinetic (PK) profiles. The precautions and methods we describe, may enable investigators to use FAERS with increased confidence and avoid confounding chemistry-based associations and reporting biases. This study also illustrates how comparative analysis of ADRs can reveal underlying mechanisms and highlight the reverse translation value in the drug discovery process. Results Analysis of content: unexpected trends in FAERS reporting The FAERS database holds over 8.5 million reports and is steadily growing (over 1,320,000 reports added in 2015; Figure 1A. We extracted 8,749,375 FAERS reports, mapped to 7,095,566 individual cases. Often a patient's condition is monitored over a span of multiple reports, which must be considered when investigating the incidence of a particular drug-ADR association (U.S. Food and Drug Administration, 2016). Figure 1 Download asset Open asset General information of the FDA Adverse Event Reporting System (FAERS) content (1997–2015). (A) The cumulative number of reports in FAERS is shown in the top panel; the bottom panel shows the number of new reports per quarter. (B) Distribution of reporter identities. Data are based on reports submitted between Q2 2002 (identification of reporting individuals started at this time) and Q4 2015. (C) Distribution of reports by the 7 ADR outcomes defined in FAERS. https://doi.org/10.7554/eLife.25818.003 Inflation of reports by multiplication can increase the apparent significance of a drug – adverse effect association, particularly when the total number of reports is low. To systematically identify the most similar cases, we compared all pairs of reports using demographic and prescription data. Almost 1% of the reports in FAERS (61,780 cases) represent multiple entry cases with identical drugs, identical ADRs, event dates, patient age and gender (Supplementary file 1). Intriguingly, only half of the reports in FAERS were submitted by healthcare professionals (Figure 1B). Over one-third of them (3.2 million) were initiated by the patients themselves and 9% were labeled 'non-specified'. Lawyers reported 3% of all FAERS cases (Figure 1B). FAERS uses seven descriptors of report outcomes: 'Death', 'Life-Threatening', 'Disability', 'Congenital Anomaly', 'Required Intervention to Prevent Permanent Impairment/Damage', 'Hospitalization – Initial or Prolonged', and 'Other'. Among these, only 'Other' is used to report relatively benign outcomes. Unexpectedly, only around 40% of the outcomes were identified as 'benign', whereas almost 15% of reported cases result in death (Figure 1C). It is a feature of reporting in an open submission database like FAERS that this ratio does not reflect the true balance between fatal and relatively benign drug ADRs, but rather the ratio of the ADRs that are thought to merit reporting. Among the 945,526 reports where death is the outcome of the ADR, 42,526 were linked to cardiac arrest and 50,155 to suicide. Top molecular ingredients of drugs that were primary suspects in death reports were rosiglitazone: 17,165 (indication type II diabetes), rofecoxib: 11,386 (primary indications: arthritis, pain; withdrawn from the clinic), reteplase: 11,386 (indication of acute myocardial infarction (MI)), and thalidomide: 17,104 (indication of myeloma multiplex; additionally, 26,429 cases of death have been attributed to lenalidomide, a derivative of thalidomide also prescribed for myeloma). For drugs like rofecoxib or rosiglitazone, which are prescribed for manageable and non-life threatening diseases, the inference that the ADR has led to death can be reasonably made. Similarly, a comparison of celecoxib (reported number of deaths: 4,066; Standardized Mortality Ratio [SMR] [Everitt and Skrondal, 2010]: 1.3) and rofecoxib, which are prescribed for the same indication, highlights the significantly higher SMR of patients taking the latter drug (SMR: 5) (Rostom et al., 2007). However, the attribution of death as an ADR of thalidomide when it is used to treat myeloma multiplex, a life-threatening, malignant disease (Singhal et al., 1999) may be hard to support; it seems likely that the 'ADR' here reflects the cancer that the drug is meant to treat. Similarly, the acute myocardial infarction that reteplase is used to treat (Wooster and Luzier, 1999) may well be the cause of many of the death ADRs with which the drug is tarred, not the drug itself. When a drug is used to treat a life-threatening disease, care is warranted in interpreting death as an ADR of that drug. Mapping drugs to their molecular ingredients improves signal retrieval In most FAERS studies, drugs are identified using RxNorm (Wang et al., 2013; Nelson et al., 2011), a set of drug synonyms supplied by the National Library of Medicine. This mapping is sufficient for the questions that may be asked of FAERS by a clinical professional, such as the safety signals for a particular drug formulation. However, products that have different identities in resources such as RxNorm share common molecular ingredients and are highly similar in their activities on molecular targets. To investigate the ADRs associated with fluoxetine, for instance, one must aggregate its 378 different synonyms. Without such aggregation, well-known fluoxetine side effects such as sexual dysfunction become statistically insignificant (four cases when only the fluoxetine drug synonym Prozac is considered; Relative Reporting Ratio [RRR] = 1.75; q-value = 1), whereas once aggregated, these ADRs stand out clearly (87 cases; RRR = 6.67; q-value = 2.56·10−96). Conversely, in its non-aggregated form, Prozac appears to have statistical significant associations with sex chromosome abnormality (one case; RRR = 2.96; q-value = 2·10−3). Aggregated, however, this association becomes insignificant (one case; RRR = 2.78; q-value = 1). For those interested in the molecular basis of drug actions and side effects, a simple way to interrogate the drugs as molecules is critical. Accordingly, we mapped the active drug ingredients in over 98% of the reports using a combination of natural language processing and multiple databases of synonyms (see Materials and methods). Not only does this value compare favorably to the 81% recognition achieved using only the synonyms alone in RxNorm, but it allowed us to look for associations drawing on standard cheminformatics-based searches. Surprisingly, of the 2729 unique ingredients identified, only 1892 were annotated as a primary suspect in at least one report; said a different way, 837 active drug ingredients had no reported ADRs whatsoever. A plot of the ingredients that were associated with ADRs shows that an exponentially decaying distribution, with 90% of the ADRs attributed to 40% of the drug ingredients (Figure 2). After correction of distribution for ADRs with q-values better than 0.05, 90% may be attributed to 46% of the investigated drugs. This ingredient mapping was used throughout subsequent analyses (see Materials and methods and Supplementary Material). Figure 2 Download asset Open asset Histograms showing the distribution of the number of ADRs that were attributed to unique ingredients. (A) All observed ingredient – ADR pairs. (B) Pairs observed below the q-value cutoff of 0.05. https://doi.org/10.7554/eLife.25818.004 As expected, mapping drugs to their active ingredients, and not simply relying on synonym aggregation, reinforced the strength of the drug-ADR signals. For example, the non-steroidal anti-inflammatory drug (NSAID) indomethacin is used to treat chronic pain and fever (MedicinesComplete, 2014). When we assessed indomethacin as an ingredient, a strong signal linked it with gastric ulcer (RRR = 10.40; q-value = 3.65·10−72), and gastric ulcer hemorrhage (RRR = 7.99; q-value = 6.78·10−18). These adverse events are known from the labels of indomethacin-containing drugs, also confirmed in World Drug Index (WDI) (Thomson Scientific, 2016). However, when we searched the trade names of the drugs in which indomethacin is used (RxNorm synonym matching), these signals were dissipated in the noise: the strongest signal for gastric ulcer decreased to RRR = 1.79, q-value = 1.00; the strongest signal for gastric ulcer hemorrhage dropped to RRR = 2.42, q-value = 1.00. Bias in ADR reporting by indication, changes in regulatory, clinical, social and legal environment Sometimes, ADRs are conflated with indications, and vice versa. An example is a report of rosiglitazone being prescribed for type two diabetes mellitus, with the ADR in the report being also diabetes mellitus (Table 1). In another report, rosiglitazone was identified as the primary suspect for congestive heart failure, as well as a therapeutic agent that was prescribed for the very same condition (Table 1). We quantified this indication bias both globally and over time. Approximately 5% of all reports for any drug describe the drug's indication as an adverse event. The number of reports in which the same ADR and indication was reported increased linearly with the increasing number of yearly reports until 2011, followed by a sudden drop (Figure 3). We could trace an FDA advisory 'refresher' presentation on guidelines of ADR reporting for clinical trials to an effective date of March 28, 2011 (Devine, 2016). This document provides clear instructions for submitters to distinguish between pre-existing conditions and ADRs and indeed may have had a significant effect on reporting quality. Figure 3 Download asset Open asset Reports with wrongly identified indications or ADRs. (A) Total number of reports in a given year where the same indication and ADR were reported. (B) Number of reports in a given year where diabetes was stated as the adverse reaction caused by 1 of ADRs with and identify two FAERS reports where the ADR is confused with the For the first rosiglitazone prescribed for diabetes is identified as the primary suspect for causing diabetes mellitus as an ADR as In the is given as the indication for rosiglitazone with the reported ADR of The reporting, where both the ADR and the indication of rosiglitazone are reported mellitus congestive mellitus We a look at the reports of rosiglitazone, where of diabetes as a side effect was attributed to the of this drug relatively until association is significant considered in the reporting of rosiglitazone 2011), with an RRR = and a q-value After this association as the and reporting of this drug, to its side effects et al., and 2007). In a simple comparison of indications and reported ADRs the bias of We these methods to investigate how reports for individual drugs over time. In we monitored the total number of reports and the incidence of adverse events reported at different time When reports are by event in FAERS, occur on the first of each and even on the first of each ADR signals show a increase (see Figures and Figure The changes in drug-ADR associations over time of reflect new populations to which the drug is Figure Download asset Open asset and time of rofecoxib FAERS reports. (A) Number of reports where rofecoxib was reported as primary represent events with a impact on the FAERS reporting of (B) Relative of all ADRs observed for ADR is by a time periods on the of this drug are by with and ADR shown here are also reported in Supplementary file (C) of those reporting rofecoxib ADRs at the reporting to with the in panel of ADRs and myocardial observed in rofecoxib reports between and Figure Download asset Open asset of FAERS reports on (A) Number of FAERS reports where celecoxib was reported as primary (B) Relative of all ADRs observed for ADR is by a time periods on the of this drug are by and ADR shown here are also reported in Supplementary file (C) number of reports where celecoxib was primary each to a The top plot all reports with celecoxib as primary In the plot on the bottom the reports in which rofecoxib was also were are with those used in panel of most reported ADRs and myocardial observed in reports. those in and that this plot will not to panel here show the ratio of the number of observed events in a given year compared to one would at while the in show a of a given ADR compared to other ADRs a given of time. of those reporting celecoxib ADRs at reporting to with the in panel Figure Download asset Open asset reports. (A) Number of FAERS reports where rosiglitazone was reported as primary (B) of all ADRs observed for ADR is by a time periods on the of this drug are by and ADR shown here are also reported in Supplementary file (C) of those reporting rosiglitazone ADRs at reporting to with the in panel of ADRs observed in rosiglitazone reports. Figure 7 Download asset Open asset The of reports. (A) Number of FAERS reports where was reported as primary (B) of all ADRs was observed for pioglitazone. ADR is by a time periods on the of this drug are by and The ADR shown here are also reported in Supplementary file (C) number of reports where was primary each to a The plot on the top of the panel shows number of individual ADRs have been and the bottom the The for cardiac have been by the of ADRs observed in reports. of those reporting ADRs at reporting to with the in panel of rosiglitazone and pioglitazone. We assessed the time of reports of rofecoxib, a anti-inflammatory drug (NSAID) that pain COX-2 (Figure events over the clinical of rofecoxib its by the FDA in A clinical study by et al. in that rofecoxib increased the risk of events et al., Introduction of for events on the labels of of in of rofecoxib from the on infarction (RRR = q-value and (RRR = q-value for a of the ADRs reported for rofecoxib from its in (Figure and the study by et al. most reports were by Between the and the of the these reports while the number of reports by After the of the the number of reports from but the to myocardial infarction and to of rofecoxib was by submitters identified themselves as (see Figure We also the time of another COX-2 celecoxib, by the FDA in just before (Figure of the of celecoxib reports shows a increase in the number of reports around the increase in use associated with the of rofecoxib (Figure the of ADR in celecoxib reports is by RRR to and myocardial infarction RRR to in a similar as in rofecoxib reports (Figure and Figure The increase of the number of reports around with the safety of celecoxib, likely a report of increased risk of events in patients used celecoxib systematically over periods of time et al., We the trends in reporting of side effects of celecoxib was affected by of rofecoxib, but the distribution of ADRs was almost identical after the of reports in which rofecoxib was as a drug (Figure of this revealed that the reports this of time were largely submitted by and while the of health professionals much below the of reports for rofecoxib (Figure These trends were confirmed by (see Materials and methods and Supplementary file which that reports of myocardial infarction were significantly with reports of celecoxib by before in Supplementary file 2). We also considered the on in as a significant event on and ADR reporting and Reports After the the FDA new labeling for not just COX-2 but for all which set new safety for the anti-inflammatory (U.S. Food and Drug Administration, 2016). was from the clinic, a after in particular as the only COX-2 in the and of the labels for other with clear to side effects that the of clinical celecoxib use. we these effects, we also that the was likely to but we were not to the of increase and 2014). This clearly demonstrates the of the of drugs in the post-marketing environment where ADR reporting

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.