How to Turn Reviewer Comments into a Targeted Evidence Search

A reviewer's comment is rarely a request for more words. It is usually a request for a specific piece of evidence that the current draft does not yet carry. The fastest way to satisfy it is to convert the comment into a narrow, answerable evidence gap, rank the gaps by risk, run a search built from eligibility criteria, and log what you found and how confident you are. This article gives a complete sequence for that work, with diagnostic checkpoints, a worked due-diligence example, a reusable checklist, and stop and escalation rules.

Most revision cycles slow down for one of three reasons. The comment is phrased as a judgment ("this claim feels overstated") rather than a question, so no one knows what to search for. The team searches the whole topic instead of the gap, so results are broad and unusable. Or the response trail is never written down, so the same gap reopens in the next round.

Cochrane's guidance on searching for and selecting studies points to the fix: plan searches from explicit eligibility criteria, use multiple sources, combine free-text terms with subject headings, document the process, deduplicate records, screen in stages, and rerun searches before publication. Those steps apply just as well to a decision memo as to a systematic review.

Read every comment and tag it by what it actually demands. Four types cover most cases:

• Evidence gap. The reviewer wants a source for a claim, or a stronger source than the one cited.
• Scope gap. The reviewer thinks a population, period, or geography is missing.
• Interpretation gap. The evidence exists but the draft's reading of it is contested.
• Presentation gap. The evidence is fine but buried, mislabeled, or missing a confidence level.

Only the first two usually require new searching. Interpretation gaps often need a re-read of sources you already hold; presentation gaps need editing, not retrieval.

Checkpoint: if you cannot write the comment as a question with a subject and a boundary ("What recent evidence links X to Y in region Z?"), it is not yet an evidence gap. Send it back for clarification before spending search time.

Turn the tagged comment into a structured question with four slots: the claim under review, the population or context, the outcome or relationship, and the time window. A comment like "the market-size claim needs support" becomes: "What published estimates exist for market size of [segment] in [region] for [year range], and what method did each use?"

This structure does two jobs. It tells you what would count as an answer, and it becomes the eligibility criteria for your search. Cochrane's recommendation to plan searches from eligibility criteria is the same discipline: decide what qualifies before you look.

Checkpoint: for each question, write one sentence describing what a satisfying answer looks like and one describing what would be off-target. If both sentences are identical, the question is too vague.

You will rarely have time to close every gap. Rank them on two axes: how much the claim moves the recommendation, and how exposed the claim is if challenged.

Checkpoint: if more than a third of your gaps land in P1, the memo's argument is probably resting on too few sources. Narrow the recommendation before searching further.

A narrow search uses the four slots from Step 2 as building blocks. Combine free-text terms for each slot with subject headings or field tags where the database supports them, which is the approach Cochrane describes. PubMed, for example, supports field tags, phrase searching, filters, and an Advanced Search, and its search details show how terms were translated and combined, so you can see whether the query did what you intended. Google Scholar lets you search by author and title, apply date limits, sort by date, and follow Cited by and Related articles to move from a known paper to adjacent ones.

A practical build order:

1. Start from the single best source you already have and use its citation trail (Cited by, Related articles, Similar Articles) to find neighbors.
2. Write one free-text string per slot, then combine them.
3. Add date limits that match your time window.
4. Run the same logic in a second source so you are not dependent on one index.
5. Save the query and its date so the search is reproducible.

For teams that want a scholarly search interface in one place, WisPaper search is the entry point for finding academic papers; the public Scholar Search page explains the same capability for readers who want an overview first.

Checkpoint: before screening results, confirm the query returns your known best source. If it does not, the query is wrong, not the literature.

Screen in two passes. First, titles and abstracts against your eligibility sentence. Second, full text against the same sentence. Cochrane recommends deduplicating records and linking multiple reports of one study, which matters because the same finding often appears in a working paper, a journal article, and a policy brief.

For every source you keep, record provenance in a fixed set of fields:

• Full citation and identifier
• Source type (peer-reviewed, working paper, official statistics, industry report)
• Date of publication and date retrieved
• The exact claim it supports, quoted or closely paraphrased
• The confidence level you assign, with a one-line reason

Use three confidence levels and define them in the memo itself: High (multiple independent sources agree, method is transparent), Medium (one solid source, or sources agree but method is partly opaque), Low (single source, indirect evidence, or method unclear).

Checkpoint: any claim marked High must have at least two independent sources in the log. If it does not, downgrade it.

A response trail only works if the next reviewer can open the sources. Save or upload the papers you rely on into My Library so the set stays organized alongside the memo. When a colleague asks a follow-up about one of those papers, Library QA lets you ask questions based on papers uploaded to My Library or added from search results, which keeps the answer tied to the same evidence set.

For a longer revision that needs a structured literature pass rather than a single query, Scholar Agent covers literature review and related longer research tasks. If the memo will be reused across cycles, set it up in Research Projects so the evidence and the response trail live together.

Checkpoint: a new team member should be able to open the project, read the gap list, and reach every cited source in under five minutes. If not, the trail is incomplete.

A team submits a due-diligence memo recommending an acquisition. The reviewer returns four comments.

Comment 1: "The claim that demand is growing 20% annually is unsupported." Tagged as an evidence gap. Rewritten: "What published estimates exist for annual demand growth in [segment] for [years], and what method did each use?" Priority P1, because the growth claim drives the valuation.

Comment 2: "You have not considered the regulatory change in the target's main market." Tagged as a scope gap. Rewritten: "What official sources describe the regulatory change and its effective date?" Priority P1.

Comment 3: "The competitor comparison feels selective." Tagged as an interpretation gap. Rewritten: "Which competitors meet the stated inclusion criteria, and did the draft apply them consistently?" Priority P2.

Comment 4: "Add confidence levels." Tagged as a presentation gap. No search needed.

For Comment 1, the team starts from the one industry report already cited, follows its citation trail, writes a query combining the segment, the region, and the year window, and runs it in two sources. They keep three estimates, log each with its method, and assign Medium confidence because the methods differ. For Comment 2, they locate the official regulatory text and assign High confidence because it is a primary source with a clear effective date. For Comment 3, they re-apply the inclusion criteria and drop one competitor that never met them, which changes the comparison table but not the recommendation.

The response trail records, for each comment: the rewritten question, the query used, the date run, the sources kept, the confidence level, and the exact sentence in the memo that changed. The reviewer can trace every edit back to a source.

Use this for each revision round.

• Every comment tagged by type (evidence, scope, interpretation, presentation)
• Every evidence and scope comment rewritten as a question with subject and boundary
• Each question has a written "satisfying answer" and "off-target" sentence
• Gaps ranked P1–P4 by decision risk and exposure
• Each search built from the question's four slots, with date limits
• Each query returns the known best source before screening
• Two-pass screening completed; duplicates and linked reports reconciled
• Provenance fields filled for every kept source
• Confidence level assigned with a one-line reason
• Every High-confidence claim backed by two independent sources
• Response trail links each comment to the query, sources, and edited sentence
• Searches rerun before the memo is finalized

Set these before you start so the team does not over-search or under-search.

Stop searching when: the question's satisfying-answer sentence is met by two independent sources; or three query variants return the same set with no new eligible records; or the remaining gap is P3/P4 and the deadline is inside one working day.

Escalate when: a P1 gap has no eligible source after a full search cycle; sources conflict on a number that drives the recommendation; the only available sources are Low confidence and the claim is P1; or a reviewer's comment cannot be rewritten as an answerable question after one clarification round.

Escalate to whom: the memo owner for scope decisions, and the reviewer for interpretation disputes. Record the escalation and its outcome in the trail so the next round starts from a known position.