Build the protocol by assigning two independent reviewers to every record, defining a conflict-resolution route, and logging each judgement with its source location in a signed appraisal document.
1. Fix the appraisal instrument and judgement scale
Choose a design-specific tool and state the judgement categories in the protocol. Cochrane's RoB 2 assesses a specific randomized-trial result across bias domains using signalling questions and written justifications, producing low risk, some concerns, or high risk judgements. NHLBI design-specific tools focus reviewers on internal validity and potential flaws in study methods or implementation, and users must define their own judgement parameters. Write those parameters into the protocol so both reviewers apply the same scale.
2. Assign two independent reviewers per record
List each record with a primary and a second reviewer. Require each to complete their judgement and source location before seeing the other's entry, so the two appraisals stay independent.
- Enter the record identifier, reviewer name, judgement, and the page, table, or section supporting it in the decision log.
- Compare the two entries only after both are complete; record agreement or the specific point of disagreement.
3. Define the conflict-resolution route
State who resolves disagreements and on what basis. A workable route: the two reviewers discuss the disputed domain and its source location; if they still differ, a named third reviewer adjudicates and the final judgement is logged with the adjudicator's name and rationale. For outcome-level certainty, GRADE provides a transparent approach to rating certainty of evidence and strength of recommendations, with explicit criteria and outcome-level judgements, so record certainty at the outcome level where your review reports it.
4. Build the decision log template
Include columns for record ID, reviewer, appraisal domain or criterion, judgement, source location, disagreement flag, adjudicator, final judgement, and date. Add a signature block for the protocol owner and a version line.
5. Sign and pilot
Sign the protocol, then run the first batch of records through the log to confirm both reviewers can cite source locations consistently before scaling up.