November 9, 2026

How to Deduplicate Search Results from Multiple Databases

When you search several databases for one review, memo, or deck, the same paper arrives more than once — as a preprint, a conference version, and a journal article, or simply as repeated records from overlapping indexes.

Written byWisPaper TeamAI Research Workflow Team
Editorial illustration for How to Deduplicate Search Results from Multiple Databases

When you search several databases for one review, memo, or deck, the same paper arrives more than once — as a preprint, a conference version, and a journal article, or simply as repeated records from overlapping indexes. Deduplication is the step that turns those overlapping exports into one defensible evidence base. The goal is not a shorter list; it is a list where every retained record has a clear identity, a documented reason for inclusion, and a traceable path back to the source that produced it.

This guide walks through the decisions and artifacts that make that possible, from planning the search to signing off on the final set. It follows the logic that Cochrane's Handbook lays out for searching and selecting studies: plan from eligibility criteria, search multiple sources, document the process, deduplicate records, screen in stages, and link multiple reports of one study.

Why do duplicates appear across databases in the first place?

Duplicates are a structural feature of multi-database searching, not a sign that someone did the search wrong. Cochrane explicitly recommends using multiple sources, which means overlap is expected. The same work can surface as a journal article, a conference paper, a preprint, or a later report, and each database may index a different combination of these.

PubMed's User Guide notes that its search details reveal how terms were translated and combined, which matters because two databases can return the same paper under slightly different metadata. Google Scholar's help pages describe features like searching by author and title, locating alternative versions of a work, and exporting citations — all of which can produce additional records for a work you already have.

The practical implication: treat deduplication as a planned stage with its own inputs and outputs, not as a cleanup you do at the end when the list feels too long.

What should exist before you start deduplicating?

Before you merge anything, you need three artifacts in place. Without them, deduplication becomes guesswork.

1. A written eligibility definition. Cochrane recommends planning searches from eligibility criteria. Write down what counts as an included study, what counts as an excluded one, and what counts as a report of an already-included study. That third category is where most deduplication confusion lives.

2. A documented search record per database. For each source, capture the date searched, the exact query, any filters applied, and the number of records returned. PubMed supports saved searches and field tags, and its search details show how terms were translated — record those details rather than reconstructing them later.

3. A stable record identifier scheme. Decide now how you will label each record (for example, a source prefix plus a sequence number) so that every row in your working file can be traced back to the export it came from.

If you are running this inside a shared workspace, set up the project container before the exports pile up. WisPaper's Research Projects area is where a team can create a project to hold the work, so the deduplication file and its provenance notes live alongside the rest of the effort rather than in scattered downloads.

How do you decide what counts as a duplicate?

This is the central judgment call, and it has two distinct answers that teams often blur together.

Record-level duplicates are the same bibliographic record appearing twice — identical title, authors, year, and source. These are mechanical to resolve.

Study-level duplicates are different records describing the same underlying work. Cochrane's guidance to link multiple reports of one study is the relevant principle here: a conference abstract and the full journal article may both be legitimate reports of one study, and you need a rule for whether you keep both, keep one as primary, or link them.

A workable decision rule has three tiers:

  • Merge when title, first author, year, and venue all match, or when a DOI or other persistent identifier matches.
  • Link when records clearly describe the same study but differ in publication type or completeness — keep both, mark one as primary, and note the relationship.
  • Keep separate when the match is only partial and you cannot confirm identity from the metadata alone.

Write this rule down before you apply it. A rule applied consistently across the whole set is defensible; a rule invented per record is not.

What does a working deduplication sequence look like?

Here is a sequence with checkpoints you can reuse. Each checkpoint is a place to stop and confirm before moving on.

Step 1: Freeze the exports

Export each database's results once, at a recorded date, and do not re-export mid-process. Save the raw files untouched. Checkpoint: every export has a matching entry in your search record with date, query, and count.

Step 2: Normalize the fields

Bring all exports into one schema: title, authors, year, venue, identifier, source database, and a record ID. Normalization is what makes comparison possible; without it, formatting differences masquerade as content differences. Checkpoint: every row has a source database and a record ID.

Step 3: Run exact-match passes

Start with the mechanical tier: exact identifier matches, then exact normalized title matches. Resolve these automatically and log what was merged. Checkpoint: the merge log accounts for every removed row.

Step 4: Run fuzzy-match review

For near-matches, review in batches rather than one at a time. Sort by normalized title so similar records sit next to each other. Apply your three-tier rule and record the decision for each cluster. Checkpoint: every cluster has a recorded decision and a reason.

For clusters you decided to link rather than merge, assign a primary record and note the relationship. Checkpoint: no study has two competing primaries.

Step 6: Screen the deduplicated set

Only now move to screening. Cochrane recommends screening in stages, and doing so after deduplication means you are not screening the same paper three times. Checkpoint: screening decisions reference the deduplicated record IDs, not the raw export rows.

Step 7: Rerun and reconcile before you publish

Cochrane recommends rerunning searches before publication. When you rerun, reconcile new results against your existing set using the same rule. Checkpoint: the reconciliation log shows what was added, what was matched to existing records, and what was excluded.

Where does a scholarly search tool fit into the sequence?

A search tool is most useful at the front of this sequence — when you are assembling the candidate set — and at the point where you need to ask questions of the material you have already kept.

For assembling candidates, WisPaper's scholarly search interface is where you find academic papers. If you want a public explanation of what that search does before you commit a team to it, the Scholar Search page describes the entry point for finding academic papers.

For longer or more involved literature work, WisPaper's Scholar Agent is designed for research tasks in its literature review, inspiration, and reproduction use cases. That is a fit when the deduplication step is part of a larger review rather than a one-off cleanup.

Once records are kept, they need a home. WisPaper's My Library lets you save or upload papers and organize them there — which is exactly what you want after deduplication, when the retained set becomes the working corpus.

And when the team needs to ask questions of that corpus, Library QA supports asking questions based on papers uploaded to My Library or added from search results. That keeps the interrogation step anchored to the set you actually deduplicated.

Worked example: a client deck with source provenance and confidence levels

Suppose a research team is preparing a client deck on a technical question. They searched three databases, exported 640 records, and need a defensible evidence base with explicit confidence levels.

The provenance table. Before deduplication, the team builds a table with one row per source database: database name, date searched, query, filters, records returned. This is the artifact that lets a reviewer reconstruct the search.

The deduplication log. After normalization, exact-match passes remove the mechanical duplicates. The fuzzy-match review produces clusters, and each cluster gets a decision: merge, link, or keep separate. The log records the record IDs involved, the decision, and the reason.

The confidence column. In the deck itself, each claim carries a confidence level tied to its evidence. A claim supported by a full journal article with a persistent identifier might be marked high confidence. A claim supported only by a conference abstract, or by a linked report that is less complete, might be marked lower. The point is that the confidence level is visible and traceable to a specific record in the deduplicated set — not asserted in prose.

The handoff note. When the deck goes to another team, the handoff includes the provenance table, the deduplication log, and the retained record list. A reviewer can follow any claim back to a record and from that record back to the database export that produced it.

This is what "ready" looks like: not a perfect list, but a list where every retained record has an identity, every removal has a reason, and every claim has a confidence level attached to a traceable source.

What should a reusable deduplication checklist contain?

A checklist keeps the process consistent across projects. Here is one you can adapt.

StageArtifactCheckpoint question
PlanningEligibility definitionIs "report of an included study" defined separately from "included study"?
SearchPer-database search recordDoes every export have a date, query, filters, and count?
NormalizationUnified schema fileDoes every row have a source database and record ID?
Exact matchingMerge logDoes the log account for every removed row?
Fuzzy matchingCluster decision logDoes every cluster have a decision and a reason?
Study linkingPrimary-record assignmentDoes any study have two competing primaries?
ScreeningScreening decisions keyed to record IDsDo decisions reference deduplicated IDs, not raw rows?
RerunReconciliation logAre new results matched, added, or excluded with reasons?
HandoffProvenance table plus logsCan a reviewer trace any claim to a source export?

Keep the checklist with the project, not in a personal folder. If the work lives in a shared project space, the Research Projects area is where a team can create and hold that project.

How do you know the deduplicated set is ready?

Readiness is a set of conditions, not a feeling. The set is ready when all of the following hold:

  1. Every retained record has a source database and a record ID. No orphan rows.
  2. Every removal is logged. The merge log and cluster decision log together account for every row that did not survive.
  3. Every study-level link has one primary. No study is represented by two competing primaries.
  4. Screening decisions reference deduplicated IDs. You can show that no paper was screened twice under different identifiers.
  5. The rerun has been reconciled. New results from the rerun are matched, added, or excluded with recorded reasons.
  6. The handoff package is complete. Provenance table, deduplication log, and retained record list travel together.

If any of these fails, the set is not ready — regardless of how clean the final list looks.

How should you handle citation integrity in the final set?

Deduplication reduces the risk of citing the same work twice, but it does not by itself confirm that every retained reference is real and correctly described. That is a separate check.

If your workflow involves BibTeX, WisPaper's TrueCite checks BibTeX against real academic databases to flag hallucinated references. Running that check on the retained set — after deduplication, before the deck or memo is finalized — catches reference problems at the point where they are cheapest to fix.

For teams that need a general entry point to the toolset, the WisPaper homepage introduces WisPaper as an AI research assistant for finding, screening, organizing, and working with academic papers. English readers can start from the English homepage.

FAQs

Title matching is a useful first pass, but it is not sufficient on its own. Titles can differ across versions of the same work, and different works can share similar titles. Use title matching as one signal alongside author, year, venue, and persistent identifiers, and route near-matches to the fuzzy-match review rather than resolving them automatically.