Original ArticleOverall bias methods and their use in sensitivity analysis of Cochrane reviews were not consistent
Introduction
Cochrane advocates rigorous methodological standards and therefore Cochrane systematic reviews are considered the gold standard when it comes to the synthesis of evidence [1]. An important part of the systematic review methodology is an appraisal of the risk of bias (RoB) in included studies. The potential effect of bias is that trialists will reach wrong conclusions about efficacy and safety of studied interventions [2].
In Cochrane systematic reviews, RoB of each included individual study is appraised using Cochrane RoB tool, which has seven domains. Cochrane authors report RoB assessment in a table, whereas they provide a judgment whether there is a “low risk,” “high risk,” or “unclear risk” of bias for each domain, and each judgment needs to be supported by the accompanying comment, which gives rationale for the judgment [3].
Authors of systematic reviews may consider making a summary assessment of RoB at a study level. However, The Cochrane Handbook Version 5.1.0 that was applied to reviews from investigated period indicates that summarizing the overall RoB in a review should be avoided because it requires value judgments about which outcomes are critical to a decision, and judgments about which outcomes are critical to a decision may vary from setting to setting. The Handbook states that “summary assessment of the risk of bias across all outcomes for a study is generally of little interest”, while it also comes with the challenge that the overall RoB might be different for different outcomes [4].
However, it has been reported that Cochrane review authors sometimes assess overall RoB and conduct sensitivity analysis based on such assessment [5]. The idea of an overall RoB is to facilitate decision making, and an overall RoB judgment was included in new tools for assessing RoB in systematic reviews [6] and nonrandomized studies about efficacy of interventions [7].
The aim of this study was to analyze definitions of “overall bias” in Cochrane reviews of interventions published within 3 years in the Cochrane Database of Systematic Reviews, as well as method and frequency of using the RoB in sensitivity analyses.
Section snippets
Study design
We conducted a primary methodological study in which unit of analysis was a published Cochrane systematic review and extracted data regarding overall RoB on a study level. Hereby we defined overall RoB on a study level as RoB assessment of an entire RCT as having an overall high, low, or unclear RoB, that is, Cochrane authors aiming to provide a single RoB judgment for an entire study, in addition to the judgment for each domain in a Cochrane RoB tool. When Cochrane authors used expression
Methods for assessing overall risk of bias in Cochrane reviews
We analyzed 1,452 Cochrane reviews (list of analyzed reviews is available in Supplementary File 2). There were 28% (N = 409) Cochrane reviews that mentioned assessment of overall RoB on the study level; 389 in the methods and 20 in other parts of the review. We divided them into 8 groups (Table 1).
Most commonly, in 131 (32%) Cochrane reviews, authors simply indicated that the overall RoB on a study level was assessed based on the criteria from the Cochrane Handbook, without any further details.
Discussion
In this study, we analyzed methodological approaches of Cochrane authors regarding an assessment of overall bias on a study level in Cochrane reviews. The main finding of our study is that the minority of Cochrane authors mention an overall RoB assessment, even fewer actually make such overall judgment, and when they do, their assessment in most of the analyzed cases was not in line with the Cochrane Handbook.
According to the Cochrane Handbook [4], authors of Cochrane reviews should avoid
CRediT authorship contribution statement
Andrija Babic: Data curation, Formal analysis, Investigation, Methodology, Visualization, Writing - original draft, Writing - review & editing. Ivana Vuka: Data curation, Formal analysis, Writing - review & editing. Frano Saric: Data curation, Formal analysis, Writing - review & editing. Ivona Proloscic: Data curation, Formal analysis, Writing - review & editing. Ema Slapnicar: Data curation, Formal analysis, Writing - review & editing. Jakica Cavar: Data curation, Formal analysis, Writing -
Acknowledgments
The authors are very thankful to Dr. Mahir Fidahić, for assistance with data extraction.
References (20)
- et al.
ROBIS: a new tool to assess risk of bias in systematic reviews was developed
J Clin Epidemiol
(2016) - et al.
In Cochrane reviews, risk of bias assessments for allocation concealment were frequently not in line with Cochrane's Handbook guidance
J Clin Epidemiol
(2019) - et al.
Risk of bias assessments for blinding of participants and personnel in Cochrane reviews were frequently inadequate
J Clin Epidemiol
(2019) - et al.
Risk of bias assessments for selective reporting were inadequate in the majority of Cochrane reviews
J Clin Epidemiol
(2019) Technology-assisted risk of bias assessment in systematic reviews requires precise definitions of risk of bias
J Clin Epidemiol
(2018)- et al.
Evidence-based medicine and the Cochrane collaboration
Bull NYU Hosp Jt Dis
(2009) Bias in clinical intervention research
Am J Epidemiol
(2006)- et al.
The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
BMJ
(2011) - et al.
Evaluation of the Cochrane tool for assessing risk of bias in randomized clinical trials: overview of published comments and analysis of user practice in Cochrane and non-Cochrane reviews
Syst Rev
(2016)
Cited by (16)
Author instructions in biomedical journals infrequently address systematic review reporting and methodology: a cross-sectional study
2024, Journal of Clinical EpidemiologyPostoperative positioning regimens in adults who undergo retinal detachment repair: A systematic review
2023, Survey of OphthalmologyCitation Excerpt :This included selection, performance, attrition, and reporting bias, as well as conflicts of interest and industry sponsorship (Table 1). For the attrition bias domain, we set a conservative threshold of 15% to help differentiate between high and low loss to follow-up.2,3 The Risk of Bias in Non-randomized Studies – of Interventions (ROBINS-I) tool was utilized to assess risk of bias for observational studies (Table 2).36
Microinvasive glaucoma surgery: A review of 3476 eyes
2021, Survey of OphthalmologyCitation Excerpt :As length of follow-up was not consistent between studies, a conservative threshold of 15% was set to differentiate between high and low loss to follow-up for the incomplete outcome data domain for the first 12 months of follow-up. The threshold was increased by 5% for every 12 months of additional follow-up thereafter.5,6 Articles that had a high risk of bias in four or more assessment categories were excluded.
Interventions for youth homelessness: A systematic review of effectiveness studies
2020, Children and Youth Services ReviewCitation Excerpt :Instead, we indicate and discuss risk of bias more generally with respect to overall evaluation designs, per the aforementioned schematic, and we discuss narratively the particular methodological concerns or limitations of specific studies. In the event of future syntheses and potential meta-analyses with specific subgroups of included studies (e.g., randomized trials or studies for specific intervention categories or outcomes), we would incorporate relevant quality or risk-of-bias appraisals into a coding and sensitivity analysis (Babic et al., 2020). Effectiveness studies were synthesized descriptively and not statistically (that is, with meta-analysis) due to the significant heterogeneity of interventions, evaluation designs, and outcome measures.
Limiting the search period in methodological studies
2020, Journal of Clinical Epidemiology
This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.
Conflict of interest: Andrija Babic, Tina Poklepovic Pericic, and Livia Puljak are volunteer members of Cochrane Croatia. Other authors have no competing interests to declare.
Authors’ contributions: A.B. contributed to data curation, formal analysis, investigation, methodology, visualization, writing–original draft and writing–review & editing. I.V., F.S., I.P., E.S., and J.C. contributed to data curation, formal analysis, and writing–review & editing. T.P.P. and L.P. contributed to conceptualization, data curation, formal analysis, investigation, methodology, visualization, writing–original draft, and writing–review & editing. D.P. contributed to investigation, methodology, visualization, and writing–review & editing. All authors approve the final version and agree to be accountable for the work.