How do I run a dual-reviewer appraisal workflow with an audit log?

Run dual appraisal by having two reviewers judge each result independently with written justifications, then resolve disagreements at the signalling-question level and record every decision, cited evidence, and resolution rule in an audit log.

RoB 2, the Cochrane risk-of-bias tool for randomized trials, assesses a specific trial result across bias domains using signalling questions and written justifications, producing a judgement of low risk, some concerns, or high risk. That structure gives the workflow its unit of disagreement: reviewers can agree on the overall judgement while differing on a signalling question, so adjudication should target the question where their reasoning diverges.

1. Define the appraisal unit and the question set before reviewers start. For RoB 2, fix the result being assessed and the signalling questions per domain. Expected result: both reviewers answer the same questions about the same result, so their records can be compared line by line.
2. Have each reviewer appraise independently and write a justification for every answer. Expected result: two separate records, each linking a judgement to the evidence cited for it.
3. Compare records at the signalling-question level, not only the final judgement. Expected result: a short list of questions where answers or cited evidence differ.
4. Adjudicate each disagreement and record the resolution rule used, such as a third reviewer's decision or a documented consensus discussion. Expected result: one resolved answer per question with the rule that settled it.
5. Log each decision with the evidence cited and the resolution rule, then derive the final domain and overall judgements from the resolved answers. Expected result: an audit trail that traces the final judgement back through each question to its evidence.

GRADE offers a comparable pattern for certainty ratings: explicit criteria and outcome-level judgements make the basis of each rating visible, which is what the audit log preserves. Keep the log at the level of individual questions and outcomes so a later reader can see why a judgement changed.