When meta-analysis goes wrong

A recent study on homeopathy shows reminds us that meta-analysis - like any study - can look superficially solid yet still produce nonsense

King Charles III: the world’s most famous fan of homeopathy.

Six years ago, the psychologist Michael Inzlicht told Slate magazine that “meta-analyses are fucked”.

It’s always stuck with me - the clash between that statement and how we’re meant to feel about meta-analyses. They’re supposed to be the highest form of evidence - a systematic review of all the studies on a particular scientific question, followed by a quantitative estimate of where the evidence points as a whole. No mere opinions, like in a narrative “review” article; no reliance on a single study. A meta-analysis tells us what we really want to know.

And indeed, if you roll out a meta-analysis in a debate over some scientific matter, it’s a powerful finishing move. That’s why those strange advocacy groups for ivermectin and hydroxychloroquine set up those (dodgy) meta-analysis websites during the pandemic - they knew the power of being able to say “meta-analytically, the evidence supports my view”.

And yet meta-analyses are fucked. Fucked, as Inzlicht said, because the scientific literature is fucked: the literature is full of bad studies, and regardless of how well you do your meta-analysis, if the studies you put into it are bad, your conclusions will be bad as well.

Since I seem to be making a habit on this site of criticising poorly-done meta-analyses—the one on breastfeeding and IQ, the one on nudges—I thought it might be fun to look into another. This one is on homeopathy, a topic so absurd that it acts as a nice reductio for any kind of scientific analysis.

It’s also a topic that’s very apt, given that our new King in the UK, Charles III, has for decades been a strong advocate for homeopathy, a patron and supporter of multiple homeopathy charities (most recently joining one in 2019), a political lobbyist for more homeopathy on the NHS, and—allegedly—a bully who successfully shut down academics who were critical of his support for alternative medicine. So, at the risk of getting thrown in the Tower, here goes…

The meta-analysis in question is of homeopathy and (spin the wheel…) ADHD. The idea is that a highly-diluted, individualised remedy—you get sulphur, I get phosphorus, someone else gets something horrendously poisonous but it doesn’t matter because of the dilution—might help children focus for longer and and avoid distraction. I found the paper via Nicolas Sommet, who posted it in a Twitter thread along with other silly meta-analyses.

It was published in June 2022 in Pediatric Research, which is part of the Springer-Nature stable, and is a perfectly mainstream journal. Take a look through the current issue and you’ll see a whole bunch of normal-looking research topics - a paper on homeopathy is a real aberration. And here’s the most interesting thing: perhaps because it’s in a serious journal, the meta-analysis appears to get lots of things right. It:

• Was pre-registered, making sure that all the criteria and rules for handling the data were set out in advance;

• Had a fairly clear set of inclusion and exclusion criteria and a nice flowchart to illustrate which studies were and weren’t analysed and why;

• Assessed the quality and “risk of bias” of each of the included studies using two separate checklists: the one used by the Cochrane Collaboration, and another one;

• Used the “GRADE guidelines” to assess the strength of the evidence;

• Included a test of the heterogeneity of the studies;

• Included a publication bias test;

• Is Open-Access, so anyone can read it.

Looks good, right? If you were just quickly skimming a meta-analysis to check its quality, you’d probably nod this one through. And this meta-analysis comes with a positive conclusion:

Individualized homeopathy showed a clinically relevant and statistically robust effect in the treatment of ADHD.

Which is complete nonsense. There are three reasons why.

The first reason it’s nonsense is obvious: because it’s homeopathy. Here’s a line you might not have expected to see in a scientific journal in 2022:

Homeopathy is a method of treatment that uses the ancient law of similars: “Let like be cured by like.”

Ah yes, the “ancient law of similars” - cited approvingly, too, and not just as an historical curiosity. What about the following?

The idea of a homeopathic treatment is believed to be an impulse for an organism that creates health and disease symptoms as an autonomous, autopoietic system.

To be honest, that doesn’t really help me understand. And indeed, the authors note that nobody really understands how the “law of similars”, or homeopathy in general, is supposed to work in a scientific sense. “However,”

using the gold standard tool of evidence-based medicine, randomized, blinded, and placebo-controlled studies and their summary in quantitative meta-analysis, we can at least determine whether they are clinically effective, even though we do not understand how a specific effect might be brought about.

Under normal circumstances, I’d agree with this. For instance, it’s still a mystery how Selective Serotonin Reuptake Inhibitor (SSRI) antidepressants have their effect, because as the recent controversy showed, it’s not at all clear that serotonin has much to do with depression. But SSRIs might still be effective, even if they don’t do what they claim to do in their name (selectively inhibit the reuptake of serotonin). We know they have lots of side effects, so they’re doing something, and if we get an overall signal from well-designed studies that they reduce depression levels, we don’t necessarily need to worry about how they’re doing it (of course, there’s much debate over the question of the effectiveness of antidepressants).

But homeopathy isn’t “under normal circumstances”. As the authors themselves say, there isn’t any of the original substance left in homeopathy - it’s all been diluted away. It can’t be having an effect through any normal physical channels and has to rely on some unknown, unsupported, cockamamie theory about water having a “memory”, and the solution becoming all the more powerful the more it’s diluted. So we can be pretty sure that any effect that’s found in this meta-analysis isn’t due to homeopathy but due to deficiencies in the studies they included.

(By the way, isn’t it nostalgic to be ragging on homeopathy? To me it feels like 2010 all over again).

Which brings us to the second reason the meta-analysis is nonsense: the studies included are low-quality. There are only six of them, so we can take a look at them individually. They all involved kids with ADHD symptoms, around the age of 5-11:

• The first one, from 2016, isn’t even a randomised controlled trial. It compared 20 kids with ADHD who got homeopathy to 10 kids with ADHD who got standard care (so, not a placebo). The meta-analysts themselves note that it has “high” risk of bias. Why was it even included?!

• The second study isn’t an RCT either, and also didn’t use a placebo control. The meta-analysts claim the study had 83 participants, but only 23 participants received homeopathy and there were 41 controls, so that’s 64 in total. At least this was random allocation, unlike the previous study, but there was no blinding. There was no difference in ADHD symptoms between the groups.

• The third study seems a lot more serious: it was a double-blind, placebo-controlled, crossover trial - that is, each set of children (31 in each group) got homeopathy and were given the placebo at different times. There was proper randomisation to group. There was a beneficial effect of homeopathy on their primary endpoint, a parent-rated measure of ADHD symptoms, at p = 0.0479 (phew!).

• In the fourth study, 22 kids were randomised to get homeopathy and 21 the placebo control. This was another double-blind study, but it was described as a “pilot study”. After 18 weeks there was no significant difference in ADHD symptoms.

• The fifth study is part of an unpublished MSc thesis. It included 30 kids, who were randomised to get homeopathic remedies or a placebo; it was double-blind. There were no significant differences in ADHD symptoms after 8 weeks.

• For the sixth study, 54 patients were randomised to get homeopathy or a placebo. It was only single-blind: the patients didn’t know whether they were getting placebo or not, but their doctors did. The meta-analysts claim this study found a truly enormous benefit of homeopathy for ADHD symptoms, of nearly 1.5 standard deviations. But I can’t see this result in the paper (which claims much more modest effects), and they don’t provide a calculation for how they worked it out. They also don’t share the data.

The meta-analysts ran an extra analysis dropping the first two studies because they weren’t randomised controlled trials, and still found an overall positive effect. But the weird thing is that they describe the remaining four studies as “double-blind” - when we know the final study on the list—the one with by far the biggest effect size—wasn’t double-blind at all. Didn’t anybody notice this? It’s in the first line of the Discussion!

And that’s the third reason for the nonsense conclusions: the meta-analytic evidence is extremely weak. Not only does it only include three fully double-blind randomised controlled trials (the third, fourth, and fifth studies above), but when you look at those three separately the evidence is borderline at best: an effect of .35 standard deviations, p = 0.04. To describe this as “statistically robust”, as the meta-analysts do, beggars belief (did none of the reviewers or editors think to pull them up on this?). It wouldn’t take much of a difference in any of the studies for this to have been non-significant - and if there’s just one other study out there with a small and/or non-significant effect, I doubt the overall effect would survive.

But wait - we’re forgetting something. There aren’t any other studies out there: the meta-analysts did a funnel plot test for publication bias, and it came up negative! This, by the way, is the test where you plot how precise a study is (usually bigger studies are more precise) by the effect size it found; if all’s well, you’d expect the studies to make a broadly symmetrical funnel shape around the average effect, with bigger (more precise) studies being closer to the average and smaller (less precise) ones varying more. But if there’s publication bias, and scientists aren’t publishing all their studies, you’d expect a chunk of that funnel to be missing.

The meta-analysts here found no evidence of asymmetry in their funnel plot. But funnel plots aren’t supposed to end the debate - indeed, they can’t provide a definitive answer in either direction. They can’t tell you for sure that there’s publication bias, because there are other reasons that you might get an asymmetric, non-funnel-shaped data distribution (this is just evidence consistent with publication bias).

They also can’t tell you for sure that there isn’t publication bias. In this case that’s a particularly important thing to say, because there are only six studies—or, as we saw above, really three—in this whole meta-analysis. A common rule of thumb is that you need about ten studies for a funnel plot to make sense (rules of thumb in statistics are often a bit silly, since context is key, but the overall point here is fine). It’s simply meaningless to put a tiny number of points into a funnel plot and conclude anything at all from the way they’re distributed.

So we have weak evidence, from poor studies, about a massively implausible—indeed impossible—treatment (apart from that, Mrs. Lincoln, how was the meta-analysis?). The very annoying thing is that, since this paper’s publication, the following statement is entirely true:

A meta-analysis in a Springer-Nature journal in 2022 concluded that homeopathic remedies were beneficial for children with ADHD.

On its own, that sounds quite convincing. And you just know that homeopaths are lining up to use it to convince well-meaning, worried parents to part with their cash to help their kids’ ADHD.

In a perfect world, meta-analyses really would be the super-convincing, definitive, show-stopping evidence we believe they are. But we’re far from that world. Even in journals that share a publisher with what’s supposed to be the world’s top scientific outlet (Nature), the quality-control is poor enough that meta-analyses will be waved through to publication with obvious errors, with clear over-claiming in the way they describe the evidence, and on scientific questions, like homeopathy, that are physically impossible and cannot have real effects. Sure, this meta-analysis superficially ticked a lot of the boxes - but even the quickest of glances shows that its conclusions are a million miles from reliable.

It’s time we downgraded meta-analyses in our mental hierarchy of evidence. At the very least we shouldn’t be particularly impressed at the mere mention of a meta-analysis, until we’ve judged each one on its merits. Despite their reputation, there’s nothing magical about meta-analyses: they’re just studies like everything else. And, like everything else in science—to a greater or lesser extent—they’re fucked.

Image Credit: Getty.