By SWAPNIL HIREMATH, MD
Archie Cochrane and the Cochrane Collaboration
Archie Cochrane was born in Scotland, educated in London (King’s College, University College and London School of Hygiene and Tropical Medicine) and worked in Cardiff, Wales. His work as a doctor during the Spanish Civil War and World War II, especially in a prisoner of war camp in Salonica, is credited with his push towards generating higher quality evidence. In his description of the clinical trial he conducted, he mentions James Lind as his hero. Ironically, that clinical trial – with weak randomization, open allocation, non-blinding of investigator or participants, and use of surrogate outcomes, would rate poorly in the Cochrane risk of bias tool.
But the scientific method of measuring stuck with him, and among many other achievements, he did perform a proper randomized clinical trial (RCT) a few decades later. He continued to be a strong supporter of RCTs and pushed for the Medical Research Council (MRC) to move from purely fundamental research towards applied clinical research. As an aside, the first proper RCT in the modern era was funded by the MRC and was published in 1950 – on the use of Streptomycin vs para-aminosalicylic acid, or a combination, in tuberculosis. Far more influential was his paper (and later book) published as part of the Rock Carling Fellowship, available here freely and worth a read. It’s where he puts forward the vision for RCTs in moving towards what he termed an ‘effective, efficient health service’.
The Cochrane Collaboration was established after Cochrane’s death, started off as a database of clinical trials put together by Iain Chalmers, and later included systematic reviews. Now there are about 50 Cochrane review groups, all over the world. Even the Cochrane logo is a forest plot (though after reading this, I dare not show it – but you can easily go and see it for yourself) from a 1990 systematic review (SR). This original systematic review is one of 12 RCTs, reporting a significant benefit in lowering respiratory distress syndrome from the use of antenatal steroids before preterm delivery. It is a classic example of how pooling underpowered studies can provide a powerful summary estimate. You can read more about the history of the Cochrane collaboration in this perspective from 3 decades ago, if you can open the paywall gates which, needless to say, is somewhat strange for a publicly funded organization. There is no monopoly on performing SRs, and they have proliferated widely since. However, Cochrane SRs are held in higher esteem. They are considered the gospel truth – in some cases, undeservedly, such as their review of hypertension.
Cochrane Hypertension Review
An example of a Cochrane SR which lost its way is this one, purportedly showing that intensive blood pressure (BP) lowering in patients with cardiovascular (CV) disease has no benefit. Interestingly, it is an update of another SR, done a few months earlier by the same authors – which the interested reader can check out for how much the data changed in 8 months to deserve a different analysis. The results of this SR, published in July 2018, are potentially practice changing. Hypertension Canada guidelines say that patients with established CV disease should be targeted for intensive BP lowering (systolic BP target < 120). The 2017 American Heart Association/American College of Cardiology (ACC/AHA) recommend a target of < 130/80 in these patients. Even the European Society of Cardiology/Hypertension in the recently announced 2018 update recommend systolic BP < 130. Is everyone wrong except Cochrane?
It is instructive to examine why the guidelines differ from the Cochrane SR. The systematic component of an SR refers to objective criteria for studies that should be included. This SR included RCTs with a sample size greater than 50, and more than 6 months follow up, which are quite reasonable and commonly used criteria. The outcomes reported had to include mortality and/or cardiovascular events, not just change in blood pressure, which is again a reasonable decision if the focus is on hard outcomes. The population of patients included, however, were those with established CV disease: ie history of myocardial infarction, stroke, chronic peripheral vascular occlusive disease, or angina pectoris. An intermediate or high cardiovascular risk was not sufficient (which was a criterion for many recent RCTs such as SPRINT, ACCORD and HOPE-3). So if a trial included a mix of these patients, then only the subgroup data of those with established CV disease as set out above would be used. See table 1 for how many patients from the individual trials made it to the SR.
Table 1: Sample Size of included trials, and the proportion included in Cochrane SR. Note that HOT and SPS3 contributed more than 50% weight overall
This matters because one if throwing away information when one is doing this – so you need a really good rationale for it. Since the sample size determines how much ‘weight’ a particular trial adds to the meta-analysis, this results in HOT (30%) and SPS3 (25%) being the heavyweights in Cochrane’s SR. Overall, both these RCTs reported no benefit in BP lowering – so the results of the Cochrane SR are not particularly surprising once this is taken into account. Was this a wise decision though?
Additionally, if you consider patients with high CV risk, and those with subclinical CV disease, not worthy of inclusion, consider the actual inclusion criteria of these trials and whether one thinks this is a clinical homogenous set (table 2).
Table 2: Summary of patient population included in the trials
More importantly, this is a comparison of two groups: lower and higher BP target. Over time, BP thresholds (and yes, this is all dichotomania) have decreased. In the 19th century, the discussion was about whether BP should be treated at all or whether it was a homeostatic mechanism. The 1964 Veterans Affairs trial changed that – though it was diastolic BP targets at that time (given that diastolic hypertension is more common in the young to middle-aged population studied then). In recent times the discussion has moved towards systolic BP. Specifically, in the elderly, the HYVET trial results suggested systolic BP lowering in the very elderly to < 150 mm Hg is beneficial. In the non-elderly, <140 mm Hg has been accepted for a while, since JNC 7, and even the controversial JNC 2014 recommendation was < 140 mm Hg. Hence in the last decade or two, the push has been towards even lower, ie whether < 130 mm Hg or < 120 mm Hg can provide even more benefits. The trials in this area have been careful about the choice of patient subpopulations and interventions in this push.
So one would expect this Cochrane SR would choose < 120 versus < 140? Or perhaps < 130 versus < 140? Umm, no it didn’t. The two groups were:
- lower blood pressure treatment target:
- systolic/diastolic ≤ 135/85 mmHg;
- mean blood pressure ≤ 102 mmHg.
- standard blood pressure treatment target:
- systolic/diastolic ≤ 140 to 160/90 to 100 mmHg;
- mean blood pressure ≤ 107 to 120 mmHg.
To belabor this point, the authors of the Cochrane SR are happy to compare RCTs that used systolic/diastolic or mean BP targets. Diastolic and mean BP targets are not used in recent RCTs, these criteria help certain older RCTs make it in. Also, their range is very narrow: systolic BP < 135 vs systolic BP < 140. Diastolic BP < 85 vs diastolic BP < 90. See table 3 below for the intervention and control arms of RCTs that got included to fulfill those eligibility criteria.
Table 3: Low and Standard BP targets as planned in the original trials
More importantly, not all these RCTs actually managed to lower the BP as desired – which is not totally surprising, but is somewhat important when one considers the outcome in some RCTs versus others. Note in particular the differences (or lack thereof) in achieved BP in the heavyweight HOT trial in table 4.
Table 4: Achieved systolic blood pressure in the included trials. Note that achieved SBP in the HOT trial in the intervention group was higher than achieved SBP in the control arm in ACCORD and SPRINT
So let’s recap: the Cochrane SR considers that is perfectly fine to lump an RCT from 1998, which targeted a diastolic BP < 85 and did not achieve a meaningful difference between groups, with a 2016 RCT which aimed for a systolic BP < 120, and did achieve a significant difference? Reminded of apples and oranges anyone?
The “no difference” result actually only applies to total mortality. When it comes to CV events, there is a significant lowering, but it is curiously stated as ‘the total number of cardiovascular events was not significantly reduced in the lower blood pressure target group compared with the standard group (RR 0.89, 95% CI 0.80 to 1.00; P= 0.044; six studies)’. Quite a head-scratcher. There is an 11% relative risk reduction in CV events, with a p-value < 0.05, which the authors state as being ‘not significantly reduced’. Perhaps the authors have chosen a different p-value threshold compared to Ronald Fisher’s beloved 0.05?
Why should any of this matter? Consider the implications of this Cochrane SR: they are not saying that individuals with low risk receive little benefit from intensive BP lowering (a previous Cochrane SR does say that). They are saying that individuals with highest risk, those with established CV disease, receive little benefit from intensive BP lowering. Not only does this have little biological plausibility, we see above that it is a methodologically unsound result. The SPRINT trial included individuals at higher CV risk (including those with established CV disease) and demonstrated a mortality benefit. The need to do a trial like SPRINT was precisely because previous RCTs like AASK or HOT were not designed to answer that question. The interventions were different and the treatment thresholds were different. ACCORD and SPS3 were planned during the same time – and with overlapping investigators, and deliberately chose eligibility criteria to answer questions in populations which are different: diabetes with ACCORD, and recent lacunar stroke with SPS3. Hence to take subgroups from all these wildly different RCTs and combine them into a meta-analysis is like using elephant’s foreskin for plastic surgery – ok, I’m exaggerating, but Cochrane is really getting under my nerves.
A common criticism of SRs, especially of the quantitative part (i.e. meta-analysis) is “garbage in – garbage out”. One is sometimes combining apples and oranges to make a smoothie instead of a fruit salad. Is there a better way out when one does want to pool many different trials and synthesize the evidence? The Cochrane SR discussed so far uses study-level summary data from each trial. One could do an individual patient level meta-analysis which has far more granular data. And the BP Trialists Collaboration has done just that. Even pre-dating SPRINT, in 2014, on the basis of individual patient data from 11 trials and 67,475 patients, they demonstrated the benefit of BP lowering in patients with a differing level of Cv risk. They reported that the relative risk reduction remains constant at different levels of CV risk, but the absolute risk lowering is greater in individuals at higher baseline CV risk. But we don’t even need individual patient-level meta-analysis. A more sensible SR, published in 2016 in the Lancet with study-level data, also reports significant benefit with intensive BP lowering – in patients with and without underlying CV disease. See a figure from that SR below:
Figure 5 from Ettehad et al, Lancet, 2016, reporting a benefit with BP lowering even in those with pre-existing CV disease
Should a systematic review always be on the top of the evidence pyramid? And is the Cochrane SR the last word? Hopefully, I have convinced you that Cochrane’s reviews aren’t as infallible as many believe.
There are subjective and qualitative decisions made when performing an SR, and one has to appraise them critically. SRs are reasonable when there are only a few small RCTs. Once a large randomized trial has been done, put more faith in the RCT – and when large RCTs are published with seemingly different results, try understanding the trials individually rather than putting them into a grinder. Sometimes studying the individual trees matter more than a bird’s eye view of the forest.
I admire Archie Cochrane. I’m disappointed by the organization named after him.
Individual RCTs included in the Cochrane Review
HOT https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(98)04311-6/abstract
AASK https://jamanetwork.com/journals/jama/fullarticle/195530
ACCORD https://www.nejm.org/doi/full/10.1056/NEJMoa1001286
SPRINT https://www.nejm.org/doi/full/10.1056/NEJMoa1511939?query=featured_home#t=articleTop
SPS3 https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(13)60852-1/abstract
PAST BP https://www.bmj.com/content/352/bmj.i708
Swapnil Hiremath is a nephrologist from Ottawa and co-creator of #NephJC. He pretends to diss systematic reviews despite having authored many. The opinions expressed here are personal and do not represent HT Canada.