Earlier this year, my colleagues and I were discussing the relationship between saturated fat and cardiovascular disease. One of us was writing an article on the topic and we were discussing an unusual trial often included in meta-analyses.
That trial is the Finnish Mental Hospital Study, a crossover study that compared patients on a control diet with a certain amount of saturated fat to patients on an intervention diet that replaced the saturated fat with polyunsaturated fats.
Here is a summary of the trial,
“A controlled intervention trial, with the purpose of testing the hypothesis that the incidence of coronary heart disease (CHD) could be decreased by the use of serum-cholesterol-lowering (SCL) diet, was carried out in 2 mental hospitals near Helsinki in 1959--71.
The subjects were hospitalized middle-aged men. One of the hospitals received the SCL diet, i.e. a diet low in saturated fats and cholesterol and relatively high in polyunsaturated fats, while the other served as the control with a normal hospital diet. Six years later the diets were reversed, and the trial was continued another 6 years.”
The study didn't just include men, it also included women and is discussed in a separate paper by the same research group.
In total, the "two studies" (really just one study) had a sample size of 818 participants (for hard CVD events), so they often weigh quite a bit in meta-analyses.
I’d like to bring attention to one particular meta-analysis published eight years ago by Mozaffarian, Micha, & Wallace, 2010. It’s one of the most cited meta-analyses on this topic, with Google Scholar indicating that it's been cited by over 900+ academic sources. Web of Science indicates that it’s been cited by 466 papers at the time of writing this post.
Clearly, it’s a well known study.
The meta-analysis of interest describes its inclusion and exclusion criteria as,
“We searched for all RCTs that randomized adults to increased total or n-6 PUFA consumption for at least 1 year without other major concomitant interventions (e.g., blood pressure or smoking control, other multiple dietary interventions, etc.), had an appropriate control group without this dietary intervention, and reported (or had obtainable from the authors) sufficient data to calculate risk estimates with standard errors for effects on occurrence of “hard” CHD events (myocardial infarction, CHD death, and/or sudden death). Studies were excluded if they were observational or otherwise nonrandomized;.”
So the authors state that the included studies must be randomized trials that are at least a year long and that they are excluding studies that are non-randomized or observational.
Here's a list of the studies they included. Note the design of the Finnish studies (Turpeinen, 1979 & Miettinen, 1983), which I'll touch upon below.
What Were the Results?
“Combining all trials, the pooled risk reduction for CHD events was 19% (RR = 0.81, 95% CI 0.70–0.95, p = 0.008)”
The 2010 meta-analysis found that replacing saturated fats in the diet with polyunsaturated fats had a notable, statistically significant reduction on CHD events. A 19% reduction is certainly, nothing to ignore and the confidence interval (CI) leans towards an effect. It seems promising as a dietary intervention. That could be a reason why the study is cited so widely. The quality of the trials included in the meta-analysis was low to moderate,
“Many of the trials had design limitations, such as single-blinding, inclusion of electrocardiographically defined clinical endpoints, or open enrollment. All trials utilized blinded endpoint assessment. Quality scores were in the modest range and relatively homogeneous: all trials had quality scores of either 2 or 3.”
“Visual inspection of the resulting funnel plot indicated some potential for publication bias (Figure S1), with a borderline Begg's test (continuity corrected p = 0.07), although such determinations are limited when the number of studies is relatively small.”
Regardless, the effects are quite interesting and worth exploring further.
What Went Wrong?
A major problem in this meta-analysis is that the two Finnish studies included in the quantitative analysis were *not* randomized. The authors made it clear with their inclusion criteria that they only wanted to include trials that were randomized.
The two Finnish Mental Hospital studies were labeled as “cluster randomized", which you can see in the table of characteristics from above. When this meta-analysis was published, several individuals were critical that a "cluster-randomized trial" was being labeled as a randomized trial, especially when there were only two clusters (two hospitals). This is a valid criticism because a cluster-randomized trial with only one cluster per condition is invalid for any between-group statistical comparisons. Brown et al., 2015 explain in this comprehensive article,
A particularly pernicious and invalid design that requires recognition is the inclusion of only one cluster per condition... Such designs are unable to support any valid analysis for an intervention effect, absent strong and untestable assumptions (11, 12). In such designs, the variation that is due to the cluster is not identifiable apart from the variation due to the condition.
A one-cluster-per-condition design is analogous to assigning one person to the treatment and one person to the control in an ordinary (nonclustered) RCT, measuring each person’s outcome multiple times, treating the multiple observations per person like independent observations, and interpreting the results like a valid RCT. In such a situation, the observations on person A can be tested as to whether they are significantly different from those on person B but cannot support an inference about the effect of treatment per se.
So it is clear that a one-cluster-per-condition design is not valid to ascertain much about the intervention. However, many individuals (if not all) failed to notice that the Finnish Mental Hospital studies were not even cluster randomized! There is no indication in any of the five published papers from these two studies that there is any randomization. You can check all five papers here:
|International Journal of Epidemiology||1983|
|American Journal of Clinical Nutrition||1968|
|International Journal of Epidemiology||1979|
Furthermore, cluster-randomized trials were not common when these studies were being conducted, which is why we should be skeptical of these being cluster-randomized trials.
Yet, these two studies were mistakenly labeled as being "cluster randomized" and therefore were included in the meta-analysis. Both of these studies contributed a total weight of 16% to the analysis.
And again, the authors found a pretty notable reduction in CVD events (RR: 0.81, 95% CI 0.70 – 0.95, p = 0.008)
Correcting the Error
So what happens to the results when you correct this mistake by removing the two studies?
Let’s open R and find out. Here's a link to the Excel file with the study data (after removing the Finnish studies) that are being loaded into R. If you'd like to reproduce the analysis on your own, you can find all the code at the bottom of this blog post. All the materials are also hosted on this repository.
As you can see above, rerunning the analysis after removing the Finnish studies results in the effect size shrinking from a 19% reduction to a 13% reduction (RR: 0.87, 95% CI 0.76 - 1.00). That’s a large difference!
If we’re concerned about statistical significance, the results are no longer significant. It's worth noting that the upper bound of the confidence interval barely contains the null value (1) and the lower bound includes a value as low as 0.76. It’s clear that the CI still seems to lean towards an effect.
Regardless of your statistical philosophy, this was a noteworthy, objective mistake. It was a mistake in labeling two studies as meeting the inclusion criteria, and correcting for this mistake leads to a substantial change in the results. Yet, this error has not been corrected for in the journal. In fact, this study has been around for eight years with no corrections or retractions.
I reached out to both the authors and the editors of PLoS, but to date, there are no updates or corrections on the article itself. Therefore, I suspect several people who read the article or cite it, are not aware that the summary effects are incorrect and that some of the studies in the analysis should not be there!
It is very important to note that correcting for the errors in this study does not lead to completely different conclusions. Although the effect is no longer statistically significant, it is still there based on the effect size and coverage of the confidence intervals. However, the effect is reduced.
Systematic reviews by other groups including Cochrane did *not* include the Finnish studies in their meta-analyses because the authors didn't believe that a "cluster randomized trial" with so few clusters (2) met the inclusion criteria for a randomized trial (also worth remembering, that there is no indication in any of the papers that this was even cluster randomized!). Some of these systematic reviews that exclude the Finnish studies still find a benefit to replacing saturated fats in the diet with polyunsaturated fats.
However, other meta-analyses have also found no benefit to replacing saturated fats with polyunsaturated fats.
Clearly, there is quite a bit of disagreement on this topic. Regardless, the meta-analysis in question still made a large error and it is a problem for the following reasons (even if the overall conclusions of it were not to change after the correction):
- Two prominent studies were misclassified
- The studies did not meet the inclusion criteria but were included
- Inclusion in the analysis leads to a substantially different effect size than without inclusion
- The meta-analysis is constantly cited and misleading other readers
I'm certainly not suggesting that errors do not happen, especially when undertaking such large, comprehensive projects. In fact, I would probably be suspicious if there were never any errors when such large projects were conducted!
However, I believe that when such errors are pointed out, they should be corrected as quickly and transparently as possible. Hopefully, the authors and the editors address this issue soon to prevent any further confusion.
Thanks for reading. If you have any thoughts, I'd love to hear them below!