Stop Abusing Scientific Studies

Discussions about politics and religion can often go haywire. I mean, why wouldn’t they? People are arguing about their beliefs with other people, and when someone suggests you’re wrong, that can be a gigantic blow to
your ego.

way too frustrating
Way too frustrating…

So, it should come as no surprise that discussions about nutrition and health can become as heated. Don’t believe me? Find a popular public post about diet on any social media platform and read the comments.

OK, what’s the problem?

Now, you shouldn’t be too concerned about the comments where people are just blatantly insulting each other because those aren’t worth paying attention to. What you should be paying attention to are discussions where studies are being referenced. These discussions can be a bit more civilized, but I’m proposing that they are more dangerous to the truth than the “veganism is stupid arguments.”

Still confused about what I’m speaking about? Here, I’ll give you an example.

Nutrition discussions in a nutshell.
Nutrition discussions in a nutshell.

Alright, you see my point now? Maybe you do, perhaps you don’t. I guess it would depend on what circles you consider yourself a part of. Anyway, I like to describe the interactions above as, “PubMed Wars.” It involves building your case by citing as much evidence as possible and spamming another person on social media with those papers.

I’ll discuss specifically why these discussions tend to be more dangerous than your average ad hominem discussions, but before I do, I want to propose something else –– these kinds of interactions aren’t just limited to people who don’t understand science and who are on social media, they also happen a lot in the academic literature.

Why science researchers are part of the problem too

If you happen to be a researcher or just read scientific papers as a hobby, you’ll know that certain parts of a paper, most often referred to as the “Introduction” and “Discussion,” often contain references to other scientific papers in the literature. Researchers try to build a case around a hypothesis by citing the research. The problem though is that some of them also engage in PubMed Wars but in a more formal way.

People don’t recognize it because they like to appeal to authority, because they’re short on time, or because they’re just not skeptical enough. Here, I’ll give you a brief example of how a scientific paper would look, when the author is partaking in the PubMed War:

Introduction

“Inflammation is an essential process of the human body. It allows the immune system to ward off infections. However, chronic inflammation is problematic and often associated with X and Y, which are bad. Current guidelines recommend the use of treatments A and B for lowering inflammation. Unfortunately, treatments A and B don’t have high-quality evidence behind them and are associated with a million side effects.
Vitamin C has shown promise as an intervention for inflammation. It interacts with these molecules which are known to be associated with inflammation and lowers their concentrations. Bob et al. found that Vitamin C administration in a buncha participants reduced biomarkers of inflammation. Rob et al. also found this and Todd et al. too! However, none of these studies looked at this population, which is what we’re doing.”

Why is this bad?

So, non-researchers and even researchers selectively choose evidence to back up their arguments with as many citations as possible, while excluding evidence that contradicts their narrative — which is known as confirmation bias.

Picking scientific studies like cherries.
Picking scientific studies like cherries.

Imagine this forest plot contains every study published on a topic.

The problem here is that there’s A LOT of published scientific literature. There are hundreds of papers being released every day, maybe even more. I don’t know. No one seems to have an accurate number. What I do know is that when people engage in PubMed Wars, they haven’t done comprehensive searches of the scientific literature to see whether there is any evidence contradicting their narrative, instead, they opened up Google or PubMed, typed in a couple of phrases, and cited the first few relevant papers. And this is bad because the papers likely aren’t accurate representations of the literature and people probably haven’t evaluated the QUALITY of these papers, instead, they’re focused on QUANTITY. Because more evidence = being right, right?

People aren’t always doing this intentionally, sometimes it’s just an unconscious behavior that happens, and I’ll often find myself even doing this after having worked on a few systematic reviews, and that’s alright, we all suffer from cognitive biases, but we have to acknowledge them if we hope even to overcome them.

Thorough searches of the literature shouldn’t only be reserved for systematic reviews and meta-analyses, they should be done when building a narrative using the published research, because if you’re going to make a claim, you need to take into account the totality of the evidence, rather than a few papers that support your outcome.

In conclusion, it’s a good idea to do some extensive research before you make a claim — that means utilizing folks who know how to do this such as medical librarians or systematic reviewers. So, the next time you feel an urge to reference a scientific study, whether it be on a Facebook discussion or in a paper, at least spend some time doing a somewhat comprehensive search of the literature.

1 thought on “Stop Abusing Scientific Studies

Leave some bamboo