Original Article
Overall bias methods and their use in sensitivity analysis of Cochrane reviews were not consistent

https://doi.org/10.1016/j.jclinepi.2019.11.008Get rights and content

Abstract

Objective

The objective of the study was to analyze methods of assessing “overall bias” in Cochrane reviews of interventions published in the Cochrane Database of Systematic Reviews and sensitivity analyses related to overall risk of bias (RoB).

Study Design and Setting

From Cochrane reviews published within 3 years, from July 2015 to June 2018, we extracted data regarding methods of judging overall bias for a single trial, as well as details regarding methods used in frequency of RoB in sensitivity analyses.

Results

Of the 1,452 analyzed Cochrane reviews, 409 mentioned assessment of overall RoB on a study level. In 107 reviews, authors clearly specified key domains that determined the overall RoB, whereas in the remaining reviews, assessment of overall bias was not in line with the Cochrane Handbook. Among 268 Cochrane reviews that had any RoB-related sensitivity analysis, in 56 (21%) reviews, the authors reported a significant change for at least one outcome compared with the initial analysis.

Conclusion

Highly heterogeneous approaches to summarizing overall RoB on a study level and using RoB for sensitivity analyses may yield inconsistent and incomparable results across Cochrane reviews.

Introduction

Cochrane advocates rigorous methodological standards and therefore Cochrane systematic reviews are considered the gold standard when it comes to the synthesis of evidence [1]. An important part of the systematic review methodology is an appraisal of the risk of bias (RoB) in included studies. The potential effect of bias is that trialists will reach wrong conclusions about efficacy and safety of studied interventions [2].

In Cochrane systematic reviews, RoB of each included individual study is appraised using Cochrane RoB tool, which has seven domains. Cochrane authors report RoB assessment in a table, whereas they provide a judgment whether there is a “low risk,” “high risk,” or “unclear risk” of bias for each domain, and each judgment needs to be supported by the accompanying comment, which gives rationale for the judgment [3].

Authors of systematic reviews may consider making a summary assessment of RoB at a study level. However, The Cochrane Handbook Version 5.1.0 that was applied to reviews from investigated period indicates that summarizing the overall RoB in a review should be avoided because it requires value judgments about which outcomes are critical to a decision, and judgments about which outcomes are critical to a decision may vary from setting to setting. The Handbook states that “summary assessment of the risk of bias across all outcomes for a study is generally of little interest”, while it also comes with the challenge that the overall RoB might be different for different outcomes [4].

However, it has been reported that Cochrane review authors sometimes assess overall RoB and conduct sensitivity analysis based on such assessment [5]. The idea of an overall RoB is to facilitate decision making, and an overall RoB judgment was included in new tools for assessing RoB in systematic reviews [6] and nonrandomized studies about efficacy of interventions [7].

The aim of this study was to analyze definitions of “overall bias” in Cochrane reviews of interventions published within 3 years in the Cochrane Database of Systematic Reviews, as well as method and frequency of using the RoB in sensitivity analyses.

Section snippets

Study design

We conducted a primary methodological study in which unit of analysis was a published Cochrane systematic review and extracted data regarding overall RoB on a study level. Hereby we defined overall RoB on a study level as RoB assessment of an entire RCT as having an overall high, low, or unclear RoB, that is, Cochrane authors aiming to provide a single RoB judgment for an entire study, in addition to the judgment for each domain in a Cochrane RoB tool. When Cochrane authors used expression

Methods for assessing overall risk of bias in Cochrane reviews

We analyzed 1,452 Cochrane reviews (list of analyzed reviews is available in Supplementary File 2). There were 28% (N = 409) Cochrane reviews that mentioned assessment of overall RoB on the study level; 389 in the methods and 20 in other parts of the review. We divided them into 8 groups (Table 1).

Most commonly, in 131 (32%) Cochrane reviews, authors simply indicated that the overall RoB on a study level was assessed based on the criteria from the Cochrane Handbook, without any further details.

Discussion

In this study, we analyzed methodological approaches of Cochrane authors regarding an assessment of overall bias on a study level in Cochrane reviews. The main finding of our study is that the minority of Cochrane authors mention an overall RoB assessment, even fewer actually make such overall judgment, and when they do, their assessment in most of the analyzed cases was not in line with the Cochrane Handbook.

According to the Cochrane Handbook [4], authors of Cochrane reviews should avoid

CRediT authorship contribution statement

Andrija Babic: Data curation, Formal analysis, Investigation, Methodology, Visualization, Writing - original draft, Writing - review & editing. Ivana Vuka: Data curation, Formal analysis, Writing - review & editing. Frano Saric: Data curation, Formal analysis, Writing - review & editing. Ivona Proloscic: Data curation, Formal analysis, Writing - review & editing. Ema Slapnicar: Data curation, Formal analysis, Writing - review & editing. Jakica Cavar: Data curation, Formal analysis, Writing -

Acknowledgments

The authors are very thankful to Dr. Mahir Fidahić, for assistance with data extraction.

References (20)

There are more references available in the full text version of this article.

Cited by (16)

  • Postoperative positioning regimens in adults who undergo retinal detachment repair: A systematic review

    2023, Survey of Ophthalmology
    Citation Excerpt :

    This included selection, performance, attrition, and reporting bias, as well as conflicts of interest and industry sponsorship (Table 1). For the attrition bias domain, we set a conservative threshold of 15% to help differentiate between high and low loss to follow-up.2,3 The Risk of Bias in Non-randomized Studies – of Interventions (ROBINS-I) tool was utilized to assess risk of bias for observational studies (Table 2).36

  • Microinvasive glaucoma surgery: A review of 3476 eyes

    2021, Survey of Ophthalmology
    Citation Excerpt :

    As length of follow-up was not consistent between studies, a conservative threshold of 15% was set to differentiate between high and low loss to follow-up for the incomplete outcome data domain for the first 12 months of follow-up. The threshold was increased by 5% for every 12 months of additional follow-up thereafter.5,6 Articles that had a high risk of bias in four or more assessment categories were excluded.

  • Interventions for youth homelessness: A systematic review of effectiveness studies

    2020, Children and Youth Services Review
    Citation Excerpt :

    Instead, we indicate and discuss risk of bias more generally with respect to overall evaluation designs, per the aforementioned schematic, and we discuss narratively the particular methodological concerns or limitations of specific studies. In the event of future syntheses and potential meta-analyses with specific subgroups of included studies (e.g., randomized trials or studies for specific intervention categories or outcomes), we would incorporate relevant quality or risk-of-bias appraisals into a coding and sensitivity analysis (Babic et al., 2020). Effectiveness studies were synthesized descriptively and not statistically (that is, with meta-analysis) due to the significant heterogeneity of interventions, evaluation designs, and outcome measures.

  • Limiting the search period in methodological studies

    2020, Journal of Clinical Epidemiology
View all citing articles on Scopus

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Conflict of interest: Andrija Babic, Tina Poklepovic Pericic, and Livia Puljak are volunteer members of Cochrane Croatia. Other authors have no competing interests to declare.

Authors’ contributions: A.B. contributed to data curation, formal analysis, investigation, methodology, visualization, writing–original draft and writing–review & editing. I.V., F.S., I.P., E.S., and J.C. contributed to data curation, formal analysis, and writing–review & editing. T.P.P. and L.P. contributed to conceptualization, data curation, formal analysis, investigation, methodology, visualization, writing–original draft, and writing–review & editing. D.P. contributed to investigation, methodology, visualization, and writing–review & editing. All authors approve the final version and agree to be accountable for the work.

View full text