You run a search for a focused clinical question, export the records, and start screening. By the twentieth title you notice the same trial appearing three times: once as the primary report, once as a follow-up, and once as a conference abstract. A few rows later, a preprint and its published version sit side by side. The count in your spreadsheet says 240 records; the number of actual studies is far smaller. If you build an evidence brief on that raw count, every downstream statement about how much evidence exists is inflated.
Duplicate records are a normal byproduct of searching more than one source, not a sign that the search was wrong. The Cochrane Handbook's guidance on searching for and selecting studies treats deduplication, staged screening, and linking multiple reports of one study as expected parts of the process. The practical task is to separate three different things that a raw result list conflates: records, reports, and studies. This article walks through diagnosing a cluttered result set, making the decisions that clean it, and knowing when the corrected deliverable is ready to circulate.
Why does a duplicate-heavy result set distort an evidence brief?
A result list counts records. An evidence brief needs to reason about studies. When one study generates several reports — a primary analysis, a longer follow-up, a subgroup paper, a conference abstract — each report can enter your list as a separate row. The Cochrane Handbook explicitly calls for linking multiple reports of one study during selection, which signals that the literature is structured this way by design.
That mismatch matters at three points in a brief:
- Volume claims. "We identified 240 records" is a statement about searching. "Twenty studies addressed the question" is a statement about evidence. Only the second belongs in a summary of findings.
- Weighting. If three reports of one trial are counted as three independent studies, that trial silently carries triple weight in a narrative synthesis.
- Applicability. A follow-up report may extend the time horizon of the same population. Treating it as a separate study can make an intervention look better replicated than it is.
The fix is not a better search string alone. It is a documented selection stage that collapses reports into studies before you interpret anything.
What does your starting state actually look like?
Before repairing anything, characterize the mess. Most duplicate-heavy exports share a few recognizable patterns.
Records that are literally the same entry
These come from overlapping database coverage or from running the same query in more than one place. They are the easiest to remove because titles, authors, and identifiers align.
Reports of the same study
These are the harder case. A primary trial report and its long-term follow-up share a population and an intervention but differ in title, journal, and sometimes author order. The Cochrane Handbook's instruction to link multiple reports of one study is aimed precisely at this pattern.
Versions of the same work
Preprints, accepted manuscripts, and final publications can all surface. Google Scholar's help documentation notes that it offers ways to locate alternative versions of a work, which reflects how commonly multiple versions circulate.
Near-matches that are genuinely different
Two trials in the same disease area with similar acronyms are not duplicates. Over-merging is as damaging as under-merging, because it can erase a real second study from your count.
A quick diagnostic: sample twenty rows and tally how many are literal repeats, how many are reports of an already-seen study, and how many are distinct. If literal repeats dominate, your problem is source overlap. If reports-of-the-same-study dominate, your problem is study-level linking, and deduplication software alone will not solve it.
Which decisions must you make before you start removing records?
Deduplication is a decision problem before it is a clerical one. Settle these four questions first, because each one changes what counts as a duplicate.
- What is your unit of analysis? If the brief summarizes studies, then two reports of one trial are one unit. If it summarizes reports, they are two. Most clinical evidence briefs use the study as the unit, consistent with the Handbook's linking step.
- What are your eligibility criteria? The Cochrane Handbook recommends planning searches from eligibility criteria. Criteria written before screening tell you whether a follow-up report qualifies on population and outcome, rather than forcing that judgment mid-stream.
- Which sources are in scope? Searching multiple sources is recommended, and it is also the main generator of overlap. Decide up front whether grey literature and preprint servers are in or out, and record the decision.
- How will you document the process? The Handbook calls for documenting the search and selection process. A deduplication log — what was merged, what was kept separate, and why — is part of that record.
Write these four decisions into a short protocol note before touching the export. It becomes the artifact you return to when a borderline case appears at row 180.
How do you build a search that produces fewer false duplicates?
Some duplication is unavoidable, but a well-constructed search reduces noise and makes later merging easier. The Cochrane Handbook recommends combining free-text terms with subject headings, and PubMed's user guide describes field tags, phrase searching, filters, and an Advanced Search that let you target author, title, and other fields precisely. Google Scholar's help documents searching by author and title as well as date limits and sorting.
A few practical moves follow from that guidance:
- Anchor on the study, not just the topic. Searching by author and by trial name or acronym, where one exists, surfaces the family of reports around a single study so you can link them deliberately.
- Use field tags to tighten recall. PubMed's field tags let you restrict a term to title or abstract, which cuts incidental matches that later look like duplicates.
- Inspect how the query was translated. PubMed's search details show how terms were combined and translated, which helps you explain why a given record appeared.
- Save the search. Saved searches in PubMed and email alerts in Google Scholar let you rerun the same query later. The Cochrane Handbook recommends rerunning searches before publication, and a saved query makes that rerun reproducible.
When you want a scholarly search interface that keeps the query and the resulting papers in one working environment, WisPaper search is the place to run it; the Scholar Search entry page explains the paper-finding starting point for readers new to the tool.
How should you deduplicate records without losing real studies?
Work in two passes, and keep them separate in your log.
Pass one: exact and near-exact matching. Remove records with identical identifiers, then identical titles with matching authors. This pass is mechanical and should be fast. Record the count removed.
Pass two: study-level linking. For each remaining record, ask whether it reports on a population and intervention you have already logged. If yes, attach it to that study entry as an additional report rather than deleting it. The Cochrane Handbook's linking step supports keeping the report while collapsing the study count.
Two safeguards protect against over-merging:
- Keep the report, merge the study. Deleting a follow-up report loses information you may need about outcomes or time horizon. Linking preserves it.
- Require a positive match on population and intervention. Similar titles are not enough. If you cannot confirm the same population and intervention, keep the records separate and flag the pair for a second look.
Screening in stages, as the Handbook recommends, fits this structure: title and abstract screening first, then full-text review where the linking decisions get confirmed against the actual reports.
Where does a working library fit into the repair?
Once records are deduplicated and linked, you need a stable place to hold the study-level set while you read. A flat export stops being useful the moment you start attaching multiple reports to one study.
This is where organizing papers in a library pays off. In My Library you can save papers from search and upload PDFs, keeping the linked reports of a study together as you read them. When you want to ask questions grounded in that collected set — for example, comparing how a primary report and its follow-up describe the same outcome — Library QA lets you query papers you have uploaded to My Library or added from search results.
For a brief that will be revisited by a team, Research Projects gives the work a project home, so the deduplicated set and the surrounding notes stay with the brief rather than in a personal download folder.
How do you judge evidence level, applicability, and indirectness in the cleaned set?
With duplicates removed, the interpretive work begins. Keep four judgments distinct, because collapsing them is how briefs overstate what the evidence shows.
Evidence level. This is a property of the study design and the question it addresses. A randomized trial and an observational study answer the same question with different strengths. Record the design for each linked study, not for each report.
Applicability. This asks whether the study population, setting, and intervention resemble the population your brief is about. A trial conducted in a different care setting may still be relevant, but the brief should say so explicitly rather than implying direct transfer.
Indirectness. This is the gap between the evidence you have and the question you asked — a different comparator, a surrogate outcome, a shorter follow-up than your question requires. A follow-up report of the same trial can reduce indirectness on time horizon while leaving other gaps intact.
Limitations. These are weaknesses within a study: attrition, unblinded assessment, small sample. They belong to the study entry, and every linked report inherits them unless a report addresses them.
A compact table with one row per study and columns for design, population, comparator, outcomes, follow-up, and limitations keeps these judgments visible. Reports attach beneath the study row rather than becoming rows of their own.
What does a corrected, ready-to-circulate deliverable contain?
A finished brief should let a reader reconstruct how you got from raw records to interpreted studies. Check for these elements before circulating:
- A search record. Sources searched, the query as run, date limits, and filters. PubMed's search details and Google Scholar's date limits make this reproducible.
- A selection flow. Records identified, duplicates removed, records screened, reports linked to studies, and studies included. The Cochrane Handbook's staged screening and linking steps map directly onto these counts.
- A study-level evidence table. One row per study, with linked reports noted beneath.
- Separate judgments. Evidence level, applicability, indirectness, and limitations each stated in their own terms.
- A rerun note. The Handbook recommends rerunning searches before publication; record the date of the final rerun and whether it changed the included set.
If a colleague can take your document, rerun the saved query, and arrive at the same study count, the deliverable is ready.
How do you verify the citations before the brief goes out?
A deduplicated set can still carry citation errors — a mistyped year, a mismatched author list, or a reference that does not resolve. Before circulation, screen the bibliography itself.
TrueCite checks BibTeX entries against real academic databases to flag hallucinated references, which is the right final gate for a reference list assembled across multiple searches. Pair it with the citation lookup features PubMed documents, so each entry can be traced to a record you actually screened.
What transfers to the next brief?
The specific duplicates change, but the workflow does not. Four habits carry over:
- Define the unit of analysis before searching. Study-level briefs need study-level linking from the start.
- Write eligibility criteria first. They resolve borderline merge decisions faster than any rule invented mid-screening.
- Log merges as you make them. A deduplication log is the difference between a defensible count and an approximate one.
- Rerun and re-verify before publication. Saved searches and a citation check close the loop.
For longer briefs that involve a full literature review, Scholar Agent supports literature review work within WisPaper's research-task area, and the WisPaper homepage gives a general introduction to the assistant for readers setting up their first project.




