For the last few years, it has been impossible to escape talk of the microbiome – the associated bacteria (and other organisms) that live in and on the human body. Overall, this attention has been a good thing, since it’s made people aware of just how bacteria-laden we are (not that everyone finds that a comfortable subject). And in some cases, particularly Irritable Bowel Syndrome, evidence is accumulating that microflora are a big part of the disease state, and might be a big part of treatment as well. It’s been known for a long time that antibiotic treatment can disrupt the gut biota, of course, with a particular case being C. difficile colitis. But it’s quite plausible that less dramatic changes in bacterial populations could also have noticeable effects.
Proving that, though, and unraveling those effects, is something else again. One of the most-referenced possibilities is a connection between gut flora and obesity, and there have a been a number of studies suggesting one. Now, though, a meta-analysis of all this work is enough to make a person wonder. In mouse models, connections have been reported between obese states and ratios of different bacterial populations, or between obesity and alpha diversity of the gut microbiome in general. But here are the authors of this new review of the data:
. . .We performed an extensive literature review of the existing studies on the microbiome and obesity and performed a meta-analysis of the studies that remained on the basis of our inclusion and exclusion criteria. By statistically pooling the data from 10 studies, we observed significant, but small, relationships between richness, evenness, and diversity and obesity status, as well as the RR of being obese based on these metrics. We also generated random forest machine learning models trained on each data set and tested on the remaining data sets. This analysis demonstrated that the ability to reliably classify individuals as obese solely on the basis of the composition of their microbiome was limited. Finally, we assessed the ability of each study to detect defined differences in alpha diversity and observed that most studies lacked the power to detect modest effect sizes. Considering that these data sets are among the largest published, it appears that most human microbiome studies lack the power to detect differences in alpha diversity.
These are very good points, and they’re the sort of issues that come up in all areas of science. Does your study have the statistical power to safely draw the conclusions that you’re drawing? Too often, the answer is “No, not really”. In biopharma research, some of the worst offenders are too-small rodent studies, of which there have been a great many published over the years, but this issue goes all the way up to the design of human clinical trials (indeed, it’s perhaps the most crucial issue there). In the case of this microbiome work, the machine learning issue is also a complication, as the quote mentions.
That illustration shows what happened when the authors went back to each study, took the stated machine-learning method from each, and applied it to the data from all the other studies. You will notice that some of the combinations flip between roughly 10% accuracy and 90% accuracy, and (from what I can see) every single study can show up as well below 50% or well above 50% accuracy, depending on whose model is used. So that statement above is certainly correct; the ability to classify individuals as obese by these methods of microbiome analysis is “limited”, and I think we can call that an example of academic understatement. I would have trouble distinguishing the overall effects shown by these re-analyses from random noise, honestly.
This says something about machine-learning models as well as about the complexities of microbiome research. The authors tried, but “it was not possible to identify factors that predictably affected model performance”, and the conclusion that has to be drawn is that none of these models can be said to be any better or more useful than any of the others, and it would appear that none of them are very useful at all. This is something to keep in mind when you hear about the marvels that can be produced through machine learning. I actually have little doubt that marvels are possible through such routes, but “garbage in, garbage out” is a law of the universe that no one has ever been able to repeal. In this case, it’s more like “not enough data in, not enough conclusions out”, but that’s an unbreakable one, too.
It’s also true, though, that the data on microbiome/obesity connections in rodent models is a lot more robust than this, and it would appear that these studies are in conflict with what we’re seeing in humans. Mind you, we can’t do the sort of wholesale manipulations in people that are done in some of the mice, but you’d still expect to see more than this. The best hypotheses that the authors offer are that obesity signatures might vary a lot more from person to person than we realize, and that just looking at diversity and taxonomy through 16S ribosome sequences is not going to be enough to tease these things out. There might be a whole host of rather different microbiome populations that end up doing similar effects and similar-looking ones that do completely different things (perhaps through metabolite formation), or it might all depend crucially on each person’s own immune fingerprint. At any rate, once again, It’s Not As Simple As You Would Have Hoped.