Personal Views: Learning from evidence based mistakes

I guess it's safe to say I'm a diehard evidence enthusiast. It's essential to get as close to the truth as you can, especially through the tangled mix of commercial and power biases that so plague health information. But if we all paid more attention to the potential adverse effects of evidence based medicine (EBM) too, maybe there would be even fewer mistakes. My first evidence based mistake came early in my involvement with EBM. When two of the member organisations of the consumer coalition I was involved with were set against each other because of a controversy, we did the evidence based thing. We turned to the systematic review to decide what stand we should take. The subject? Bromocriptine for lactation suppression. One of our member groups demanded that we support a call for a ban of the drug for this indication, while another objected just as vehemently to removal of access. The systematic review concluded that the drug was effective, with no concerning adverse effects. We didn't support the call for a ban, and we were wrong. The drug, it turned out, was causing serious harm, including deaths. A promising treatment is just the larval stage of a disappointing one It was the first of many experiences of being led down the garden path by a systematic review because of absent or weak information on adverse effects. Typically, trials are powered for effectiveness and are fairly short term. This is not much help for determining adverse effects if they aren't common and the trials aren't very large. EBM has a long way to go before this problem is solved, although people are working on it. Jumping to conclusions too soon is another cause of evidence based mistakes. This is a big bugbear of the leading experts in EBM, of course, and gets talked about quite a bit. But the message isn't getting through. Too many systematic reviews make judgment calls far too soon. This applies especially to reviews that speak of a “promising treatment”—a pure piece of emotionally exploitive marketing terminology if ever there was one; it should have no place in science. A promising treatment, I've learnt, is generally just the larval stage of a disappointing one. Eventually, of course, EBM is self correcting. It is science, after all. Meanwhile, it can lead people astray. In 2001, for example, I fell for this problem of believing a too early conclusion yet again. A review came out on whiplash injury, which concluded that maybe “rest makes rusty” and perhaps we should think of holding off on those neck collars. An update in 2003, though, took it back the other way: neck collars may be the way to go after all. Bad luck for people with whiplash in the care of avid EBM enthusiasts between 2001 and 2003. At least that time the harm was only sore necks. A few times a year, though, the reversal of evidence fortunes is about something life threatening. Change in practice led by EBM enthusiasts often does a lot of good. But sometimes it causes harm, especially when people react every time that individual trial results become available. Recently I saw some data comparing the practice of hospitals in Canada that had participated in a multicentre international trial of carotid endarterectomy (surgery for blocked arteries in the neck) with hospitals that had not participated. The data were put forward as proof that getting into the heart of EBM and participating in trials was an effective way to implement research results. The trial enthusiasts and the other hospitals ended up with much the same levels of intervention. However, one group had spiked precipitously up and down as positive and negative trial results were published, while the other group of hospitals had moved slowly and steadily to the same point. It is perhaps an article of faith, more than a matter of evidence, that the people being cared for by EBM enthusiasts are always better served. A great characteristic of the EBM movement that attracts many of us is its critical nature and constant concern with improving methods. I wish, though, that the movement would ponder more explicitly the adverse effects of EBM itself and the way it presents itself. EBM is a challenge to those with much money and power to lose by its advance. So I suppose it is understandable that many people in the movement focus on promotion and have a tendency to get defensive. But there's an excess of certainty, too—even some arrogance and snobbery about how the ordinary folk do things, with their attention to the evidence of their own eyes and to what others they respect are doing. This attitude can get obnoxious and is itself causing adverse effects. It limits the spread of EBM. One of the consequences of hubris is that people aren't as keenly attuned to their own mistakes as they are to the errors of others. Over time EBM should cause fewer mistakes than other options, especially profit driven medicine. The trouble is that people get hurt by the evidence based mistakes too—sometimes badly. We should be paying more attention.

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.