How do I define the judgement parameters before using a quality assessment tool?

Define judgement parameters by writing a codebook that states, for each appraisal item, what evidence counts as adequate and which judgement it triggers before you read any studies. The NHLBI design-specific tools focus reviewers on internal validity and potential flaws in study methods or implementation, and they require users to define their own judgement parameters. Cochrane's RoB 2 shows the structure to copy: signalling questions, written justifications, and a final judgement of low risk, some concerns, or high risk for a specific randomized-trial result.

Build the codebook in order

1. Fix the unit of judgement. For RoB 2, the unit is a specific randomized-trial result, so record which outcome and time point each judgement covers. For NHLBI tools, match the tool to the study design before writing any item rules.
2. List every item and its signalling questions. Copy the tool's wording exactly so reviewers answer the same prompt.
3. Write a decision rule for each item. State the evidence that counts as adequate, the evidence that counts as inadequate, and the judgement each produces. For a follow-up item, specify the minimum duration, the acceptable loss to follow-up, and what happens when a study reports neither.
4. Add a tie-break rule for partial evidence. Say whether missing information yields "some concerns" or a request to the authors, and record that choice.
5. Pilot the codebook on two or three studies, then revise ambiguous rules before full appraisal.

What a usable rule looks like

A rule such as "adequate follow-up means the study reports follow-up duration and the proportion of participants with outcome data" gives two reviewers the same check. Each reviewer can mark met, not met, or unclear and cite the page or table that supports the mark. RoB 2 pairs each judgement with a written justification, so require the same citation for every item.

Keep judgements consistent

Have two reviewers independently apply the codebook to the same study and compare item-level results. Disagreements point to a rule that needs sharper wording; revise the codebook and reapply it to the affected studies. Store the final codebook with the review so readers can see how each judgement was reached.