Two Reviewers Disagree on Which Studies to Include: Repairing a Screening Decision Before the Deadline

You have a course paper due Friday, a rubric that rewards a defensible method section, and two reviewers who reached opposite verdicts on the same stack of abstracts. One says a study belongs in the review; the other says it does not. The disagreement is not a personality clash. It is a signal that your eligibility criteria are underspecified, and it will surface again in the write-up unless you repair the process now.

This article walks through a concrete repair: revisit the criteria, calibrate on examples, record reasons, resolve ambiguous cases, and preserve an audit trail. It ends with a submission decision you can defend to a supervisor.

Cochrane guidance treats eligibility criteria as the foundation of the search itself: searches should be planned from the criteria, and the selection process should be documented. When two reviewers apply the same written criteria and reach different verdicts, the criteria are doing less work than the method section claims. That gap is what a marker or supervisor will probe.

A disagreement also threatens your timeline. If you cannot say why a record was excluded, you cannot write a clean screening paragraph, and you cannot rerun the search before submission with confidence that the same records will be handled the same way.

Before you arbitrate, name the problem precisely. In a typical student review, the starting evidence problem looks like this:

• The criteria say "empirical studies of the intervention" without defining what counts as empirical or which intervention variants qualify.
• Records were screened in one pass, so title-level and full-text judgments are mixed together.
• Multiple reports of the same study were treated as separate studies.
• Exclusion reasons were kept in a reviewer's head rather than in a log.

Cochrane recommends screening in stages and linking multiple reports of one study. If your process skipped either step, the disagreement is likely a symptom of the process rather than a genuine scientific dispute.

Rewrite each criterion as a test a second reviewer could apply without asking you a question. A useful pattern is to state the population, the intervention or exposure, the comparison, the outcome, and the study design, then add one worked example of an included record and one of an excluded record.

Two practical checks help:

1. Design boundary. If the criterion says "empirical," say whether qualitative, modelling, and secondary analyses qualify.
2. Report boundary. If a study has a conference abstract and a full paper, decide in advance that they are one study, not two.

Cochrane's emphasis on linking multiple reports of one study belongs in the criteria themselves, not only in the results narrative.

Calibration means both reviewers apply the revised criteria to a shared set of records and compare verdicts before screening the rest. Choose a small calibration set that includes at least one clear include, one clear exclude, and one genuinely ambiguous record. Discuss the ambiguous one until you can point to the specific criterion that decides it.

If the calibration set still produces disagreement, the criterion is still too loose. Tighten it and recalibrate. This loop is faster than arbitrating hundreds of records one at a time.

When you need to rebuild the candidate pool from the revised criteria, use a scholarly search interface to run the terms again rather than relying on the old list. WisPaper's scholarly search is the place to find academic papers through its search interface; the Scholar Search entry page explains the public entry point for finding papers. Keep the search strings and dates in your log so the rerun is reproducible.

A screening log with a reason code per record is the single most useful artifact you can produce under deadline. At minimum, record:

• the record identifier;
• the reviewer;
• the verdict;
• the criterion that decided it;
• the date.

Cochrane recommends documenting the selection process, and a reason-coded log is the most direct way to do that. It also converts a disagreement into a specific question: which criterion did each reviewer apply?

Ambiguity is normal at the boundary. The repair is to write a rule that resolves the class of cases, not just the single record. For example: "Where a record reports only a protocol with no results, exclude at full text and record the reason as 'no results reported.'" Once the rule exists, apply it to every similar record and note the rule identifier in the log.

If the ambiguity concerns whether two reports describe one study, resolve it by linking them and counting them once. Cochrane's guidance on linking multiple reports supports this directly.

An audit trail is the difference between "we screened the records" and "here is how we screened them." Build it from four pieces:

1. The criteria document, with version numbers and dates.
2. The calibration notes, showing which examples were discussed and which criterion decided each.
3. The screening log, with reason codes and reviewer names.
4. The search record, including strings, sources, and the date of any rerun.

Cochrane recommends using multiple sources, combining free-text terms with subject headings, deduplicating records, and rerunning searches before publication. Each of those steps should leave a trace in the audit trail.

If you are organizing the retained papers, save or upload them to a library so the included set is separate from the screened-out set. WisPaper's My Library is where you can save or upload papers and organize them. Keeping the included set in one place makes the final count auditable.

Database interfaces expose the mechanics of a search, and those mechanics belong in your log. PubMed supports field tags, phrase searching, filters, Advanced Search, saved searches, and search details that reveal how terms were translated and combined. Google Scholar documents searching by author and title, date limits, and citation export. Recording which of these you used turns a vague methods paragraph into a reproducible one.

When you need to check that your reference list matches real records, a citation-screening step helps. WisPaper's TrueCite checks BibTeX against real academic databases to flag hallucinated references. That is a reference-integrity step, distinct from eligibility screening, and it belongs after the included set is settled.

Consider a student reviewing studies of a teaching intervention for a course paper. The rubric rewards a clear method section. Two reviewers disagree on four records.

Move 1 — Name the problem. The criteria say "empirical studies" but do not define design boundaries, and multiple reports were counted separately.

Move 2 — Rewrite the criteria. Add design boundaries and a rule that multiple reports of one study count once.

Move 3 — Calibrate. Both reviewers apply the revised criteria to a shared set of ten records, including one ambiguous record. They agree on nine and write a rule for the tenth.

Move 4 — Log reasons. Each excluded record gets a reason code tied to a criterion. The four disputed records are re-adjudicated against the written rule.

Move 5 — Rerun and record. The search is rerun before submission, deduplicated, and logged with the date. The included set is saved in one library location.

At the end, the student can write a method paragraph that states the criteria, the calibration step, the reason codes, and the rerun date. That paragraph is what the rubric is asking for.

The corrected deliverable is not just a shorter list of studies. It is a criteria document, a calibration note, a reason-coded screening log, and a search record, all consistent with each other. The included set is the output of that process, not a separate judgment.

With those four artifacts in hand, the submission decision becomes concrete: submit the paper with the repaired method section, and attach the screening log as an appendix if the rubric allows. If the supervisor asks why a record was excluded, you can point to the criterion and the rule. If the supervisor asks whether the search is current, you can point to the rerun date.

For longer follow-on work, such as a thesis chapter that extends the review, WisPaper's Scholar Agent supports longer research tasks in its literature review, inspiration, or reproduction use cases. Move into that work after the screening repair has produced a stable, documented study set.

• Write criteria as tests, not themes.
• Calibrate before screening, not after disagreement.
• Record a reason for every exclusion, every time.
• Resolve ambiguity with a written rule that covers the class of cases.
• Rerun the search before submission and log the date.

These lessons transfer to any review with more than one reviewer, and they transfer to solo reviews where the second reviewer is a later version of yourself.