Objective: To review the findings of studies that have evaluated the design and/or usability of key risk of bias (RoB) tools for the assessment of RoB in primary studies, as categorised by the LATITUDES Network (a searchable library of RoB tools for evidence synthesis): PROBAST, RoB2, ROBINS-I, QUADAS-2, QUADAS-C, QUAPAS, ROBINS-E, and the COSMIN RoB checklist. Study design and setting: Systematic review of methodological studies. We conducted a forward citation search from the primary report of each tool, to identify primary studies aiming to evaluate the design and/or usability of the tool. Two reviewers assessed studies for inclusion. We extracted tool features into Microsoft Word and used NVivo for document analysis, comprising a mix of deductive and inductive approaches. We summarised findings within each tool and explored common findings across tools. Results: We identified 13 tool evaluations meeting our inclusion criteria: PROBAST (3); RoB2 (3); ROBINS-I (4); QUADAS-2 (3). We identified no evaluations for the other tools. Evaluations varied in clinical topic area, methodology, approach to bias assessment and tool user background. Some had limitations affecting generalisability. We identified common findings across tools for 6/14 themes: 1) challenging items (e.g. RoB2/ROBINS-I "deviations from intended interventions" domain), 2) overall RoB judgement (concerns with overall risk calculation in PROBAST/ROBINS-I), 3) tool usability (concerns about complexity), 4) time to complete tool (varying demands on time e.g. depending on number of outcomes assessed), 5) user agreement (varied across tools), and 6) recommendations for future use (e.g. piloting) and development (add intermediate domain answer to QUADAS-2/PROBAST; provide clearer guidance for all tools). Of the other eight themes, seven only had findings for the QUADAS-2 tool, limiting comparison across tools, and one ("re-organisation of questions") had no findings. Conclusion: Evaluations of key RoB tools have posited common challenges and recommendations for tool use and development. These findings may be helpful to people using or developing RoB tools. Guidance is necessary to support the design and implementation of future RoB tool evaluations.
Common challenges and suggestions for risk of bias tool development: a systematic review of methodological studies / Tomlinson, Eve; Cooper, Chris; Davenport, Clare; Rutjes, A; Leeflang, Mariska; Mallett, Sue; Whiting, Penny. - In: JOURNAL OF CLINICAL EPIDEMIOLOGY. - ISSN 0895-4356. - 171:(2024), pp. 1-21. [10.1016/j.jclinepi.2024.111370]
Common challenges and suggestions for risk of bias tool development: a systematic review of methodological studies
Rutjes AConceptualization
;
2024
Abstract
Objective: To review the findings of studies that have evaluated the design and/or usability of key risk of bias (RoB) tools for the assessment of RoB in primary studies, as categorised by the LATITUDES Network (a searchable library of RoB tools for evidence synthesis): PROBAST, RoB2, ROBINS-I, QUADAS-2, QUADAS-C, QUAPAS, ROBINS-E, and the COSMIN RoB checklist. Study design and setting: Systematic review of methodological studies. We conducted a forward citation search from the primary report of each tool, to identify primary studies aiming to evaluate the design and/or usability of the tool. Two reviewers assessed studies for inclusion. We extracted tool features into Microsoft Word and used NVivo for document analysis, comprising a mix of deductive and inductive approaches. We summarised findings within each tool and explored common findings across tools. Results: We identified 13 tool evaluations meeting our inclusion criteria: PROBAST (3); RoB2 (3); ROBINS-I (4); QUADAS-2 (3). We identified no evaluations for the other tools. Evaluations varied in clinical topic area, methodology, approach to bias assessment and tool user background. Some had limitations affecting generalisability. We identified common findings across tools for 6/14 themes: 1) challenging items (e.g. RoB2/ROBINS-I "deviations from intended interventions" domain), 2) overall RoB judgement (concerns with overall risk calculation in PROBAST/ROBINS-I), 3) tool usability (concerns about complexity), 4) time to complete tool (varying demands on time e.g. depending on number of outcomes assessed), 5) user agreement (varied across tools), and 6) recommendations for future use (e.g. piloting) and development (add intermediate domain answer to QUADAS-2/PROBAST; provide clearer guidance for all tools). Of the other eight themes, seven only had findings for the QUADAS-2 tool, limiting comparison across tools, and one ("re-organisation of questions") had no findings. Conclusion: Evaluations of key RoB tools have posited common challenges and recommendations for tool use and development. These findings may be helpful to people using or developing RoB tools. Guidance is necessary to support the design and implementation of future RoB tool evaluations.File | Dimensione | Formato | |
---|---|---|---|
TomlinsonRutjes2014_preprint.pdf
embargo fino al 17/04/2025
Descrizione: This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of record.
Tipologia:
Versione dell'autore revisionata e accettata per la pubblicazione
Dimensione
841.08 kB
Formato
Adobe PDF
|
841.08 kB | Adobe PDF | Visualizza/Apri Richiedi una copia |
Pubblicazioni consigliate
I metadati presenti in IRIS UNIMORE sono rilasciati con licenza Creative Commons CC0 1.0 Universal, mentre i file delle pubblicazioni sono rilasciati con licenza Attribuzione 4.0 Internazionale (CC BY 4.0), salvo diversa indicazione.
In caso di violazione di copyright, contattare Supporto Iris