Audit an extraction log by sampling fields, comparing each against its source page, and recording whether the field is traceable, correct, or needs re-extraction. The audit log then reports a pass rate and applies an escalation rule to untraceable fields.
1. Define the fields and the trace target
List every extracted field (for example, sample size, outcome definition, or risk-of-bias judgement) and the exact page, table, or section where its value should appear. A field with no stated source location cannot be checked, so mark it untraceable at this stage.
2. Sample and compare against the source
- Draw a sample of extracted fields across studies and field types, and open the cited source page for each one.
- Record one status per field: traceable and correct, traceable but incorrect, or untraceable because the value cannot be located at the cited location.
- Note the source location you checked so a second reviewer can repeat the comparison.
3. Compute the pass rate
Report the pass rate as the share of sampled fields that are both traceable and correct. Keep incorrect and untraceable fields as separate counts, because they call for different fixes: incorrect values need correction, while untraceable values need re-extraction from the source.
4. Apply an escalation rule
Set the rule before auditing. A workable example: any untraceable field is escalated to a second reviewer who re-extracts it from the source page; if the re-extracted value differs from the log, the field is corrected and the change recorded. Escalation should also cover fields whose cited location does not contain the reported value.
5. Record the decisions
For each escalated field, log the reviewer, the source location checked, the re-extracted value, and the final status. This creates an audit trail that shows how each decision was reached.
Where structured appraisal helps
Cochrane's RoB 2 approach assesses a specific randomized-trial result across bias domains using signalling questions and written justifications, producing low risk, some concerns, or high risk judgements. That structure is a useful model for extraction audits: each judgement carries a written reason tied to the source. PRISMA 2020 provides a checklist and flow diagrams for transparent reporting of why a review was done, what methods were used, and what studies were found and included, which can frame how the audit is reported.
Reference checks
TrueCite checks BibTeX against real academic databases to flag references that need verification. Use it to identify records for manual verification against the cited source.