Assign two independent reviewers to each trial result, require written signalling-question justifications, and route disagreements to criterion-level adjudication. RoB 2 assesses a specific randomized-trial result across bias domains using signalling questions and written justifications, producing low risk, some concerns, or high risk judgements (Cochrane Handbook, Chapter 8).
Set up the unit of assessment
Define each row of your dataset as one trial result, since RoB 2 applies to a specific result rather than to a study as a whole. Record the trial identifier, the outcome, and the result being judged so both reviewers assess the same target.
Run the workflow
- Assign two reviewers to each trial result and have each complete the RoB 2 domains independently, answering every signalling question and writing a justification for each answer. Expected result: two complete, separately justified assessments per result.
- Compare the two assessments at the domain level and flag any result where the domain judgements differ. Expected result: a list of divergent results with the specific domains and signalling questions in dispute.
- Route each flagged result to a criterion-level adjudication step where a third reviewer or the pair resolves the disagreement question by question, citing the trial report. Expected result: a reconciled judgement for each disputed domain.
- Record every change in a reconciliation log that names the result, the domain, the original judgements, the resolution, and the reason. Expected result: an auditable trail from initial divergence to final judgement.
- Summarize resolved divergences by domain and by direction to show where reviewers most often disagreed. Expected result: a short summary that accompanies the completed RoB 2 dataset.
Keep judgements consistent across many trials
Agree on judgement parameters before starting, because design-specific quality tools require users to define their own thresholds and criteria (NHLBI Study Quality Assessment Tools). A short calibration pass on a few trial results lets reviewers align on how they apply signalling questions before the full set begins.
When the completed dataset feeds a meta-analysis, carry the domain judgements into sensitivity analyses so readers can see how pooled results respond to excluding high-risk results (Cochrane Handbook, Chapter 10).