How to Prepare the Literature Check for a Replication Study

A replication study stands or falls on whether you can show that you knew what the original study claimed, what later work said about it, and where your search stopped. The literature check is the part of the project that produces that record. It is not a single search box session; it is a planned, documented sequence that starts from your eligibility criteria and ends with a search log a coauthor or reviewer can audit.

This guide walks through that sequence for a manuscript, a coauthor handoff, or a peer-review response. It assumes you are either revising a replication manuscript, closing a project, or starting a new review, and that you need an evidence trail rather than a reading list.

The Cochrane Handbook's guidance on searching for and selecting studies gives a useful spine: plan searches from eligibility criteria, use multiple sources, combine free-text terms with subject headings, document the process, deduplicate records, screen in stages, link multiple reports of one study, and rerun searches before publication. Each of those is an artifact you can hand to someone else.

For a replication study, the eligibility criteria have a specific shape. You need to define:

• The original study you are replicating, including all its reports.
• Direct replications and failed replications of that study.
• Conceptual replications that test the same claim with different materials or samples.
• Critiques, commentaries, and corrections that change how the original should be read.
• Methodological papers that define the procedure you are following.

Write these as inclusion and exclusion rules before you search. A rule like "include any study testing the same effect with the same manipulation" is searchable; "include relevant work" is not.

Most replication literature checks go wrong at the start, not the end. Sort yourself into one of these states and act accordingly.

You have the original paper only. You need a seed set. Pull the original's reference list, its citing works, and any registered replication attempts. Build your term list from the original's own vocabulary.

You have a partial reading list from a coauthor. You need to convert it into a reproducible search. Extract the terms and sources behind each item, then check whether the list covers replications, critiques, and corrections, not just supporting citations.

You are revising after peer review. You need to rerun the search and document what changed since submission. Reviewers often ask whether new work appeared; your log should answer that with dates.

You are handing off to a new lab member. You need the search strategy, the deduplicated record set, the screening decisions, and the reasons for exclusions, so the next person can continue without redoing the work.

Translate each eligibility rule into terms. Cochrane's guidance to combine free-text terms with subject headings applies directly here: the original authors may have used a label that differs from the one later replicators use, so you need both controlled vocabulary and natural-language variants.

PubMed's documentation is helpful for making this concrete. It supports field tags, phrase searching, filters, Advanced Search, saved searches, Similar Articles, citation lookup, and links to full text when available, and its search details reveal how terms were translated and combined. That last point matters for auditability: you can record not just what you typed but how the database interpreted it.

A practical term-building step:

1. List the construct names, synonyms, and abbreviations used across the original and its replications.
2. Add the method terms (task name, manipulation, measure).
3. Add study-type terms for replication, direct replication, conceptual replication, and failed replication.
4. Run each block separately, then combine, and save the combined strategy.

Use WisPaper search to run scholarly searches and collect candidate papers in one place, and the WisPaper Scholar Search entry point if you want a public explanation of the search workflow before you begin. Neither replaces your term list; they are where the list gets executed.

Cochrane recommends using multiple sources, and for replication work the sources serve different jobs. A general scholarly index finds cross-disciplinary replications; a biomedical database finds indexed method papers; a citation index finds who cited the original. Google Scholar's help pages document searching by author and title, date sorting and date limits, Cited by and Related articles, citation export, library access links, email alerts, and ways to locate alternative versions of a work. Those features map onto specific steps: Cited by for forward citation chasing, Related articles for adjacent work, and alerts for the rerun before publication.

Keep a source log with one row per source: database or tool, date run, exact strategy, filters applied, and number of records returned. This is the row that makes your method auditable.

Cochrane's sequence is deduplicate records, then screen in stages. Do the same. After exporting records, remove duplicates and assign each remaining record a stable identifier. Then screen in two passes: title and abstract first, full text second.

At each pass, record a decision and a reason. A screening table with columns for record ID, stage, decision, reason, and reviewer is enough. When two people screen, note disagreements and how they were resolved.

This is also where a library pays off. Save or upload the records you keep into My Library so the screened set, not just the raw exports, is the thing you and your coauthors work from. If you need to ask questions across that saved set, Library QA lets you query papers uploaded to My Library or added from search results, which is useful when you are checking whether a candidate replication matches your inclusion rule.

A single study often appears as a preprint, a conference abstract, and a journal article, sometimes with different author orders or titles. Cochrane's guidance to link multiple reports of one study exists precisely to stop these from being counted as separate evidence. In a replication check, this step also prevents double-counting a replication that was published twice.

Build a linking table: study ID, report IDs, and the relationship (preprint of, published version of, correction to, commentary on). When you cite the replication in your manuscript, cite the version of record and note the linked reports in your methods appendix.

A replication manuscript is unusually exposed to citation problems because it cites the original, its replications, and its critiques, often pulled from many sources. Before references go into the draft, screen the BibTeX. TrueCite checks BibTeX against real academic databases to flag hallucinated references, which is the concrete step for catching entries that were assembled from memory or from a secondary source rather than from the paper itself.

Run this check after the linking table is stable and before the reference list is finalized, so you are verifying the set you will actually cite.

Suppose a lab is revising a replication of a published effect and a new coauthor is taking over the literature section. The handoff package should contain:

• Eligibility rules for original reports, replications, critiques, and method papers.
• Term blocks with the construct names and method terms, plus the combined strategy.
• Source log with dates, strategies, filters, and record counts.
• Deduplicated record set with stable IDs.
• Screening table with decisions and reasons at both stages.
• Linking table connecting reports of the same study.
• Citation check results for the BibTeX that will appear in the manuscript.

The new coauthor can then rerun the saved strategies, compare record counts to the log, and see exactly which records were excluded and why. If a reviewer asks whether new replications appeared since submission, the rerun answers it with a dated comparison rather than an impression.

For longer tasks that span the whole review, Scholar Agent fits literature review and reproduction use cases, and if the replication involves re-running an analysis from the original paper, PaperClaw supports reproduction planning from an uploaded PDF and reproduction instructions. Keep the project's artifacts together in Research Projects so the search log, screening table, and manuscript notes live in one place.

Use these to decide when the check is done and when to bring someone else in.

Stop searching when a rerun of the saved strategies returns no new records that meet the eligibility rules, and your source log shows every planned source has been run at least once with a recorded date.

Escalate to a coauthor when two screeners disagree on a record that would change the replication's framing, or when a candidate replication's method differs enough that you cannot decide whether it is direct or conceptual.

Escalate to a librarian or methods reviewer when your term blocks return very different counts across sources and you cannot explain the gap from the search details.

Pause the manuscript when the citation check flags an entry you cannot resolve to a record, or when the linking table shows a study counted twice.

The literature check is ready when a reader who was not involved can reconstruct your search from the artifacts alone. Concretely: the eligibility rules are written, the term blocks and combined strategies are saved, the source log is dated and complete, the record set is deduplicated, the screening table has reasons for every exclusion, the linking table counts each study once, the citation check has been run on the final BibTeX, and the search has been rerun before submission with the comparison recorded.

That standard is the same one Cochrane describes for systematic searches, scaled to a replication study. It is also what makes the check reusable: the next replication in your lab starts from a documented method rather than from scratch.