How to Remove Duplicate References Without Losing Distinct Studies

Duplicate removal in a review is not a single "delete duplicates" click. It is a sequence of decisions that separates three different things: the same record arriving twice from different sources, multiple reports of one study (a primary paper plus follow-up or secondary analyses), and genuinely distinct studies that happen to share authors, cohorts, or titles. Getting this wrong in either direction is costly — you either double-count a study in synthesis or silently drop a report that carries outcome data you need.

This article gives a practical, checkpoint-driven sequence for clinicians and researchers working on a focused evidence question. The running example is a review of a single population and outcome: adults with community-acquired pneumonia, asking whether a specific adjunctive therapy reduces 30-day mortality. The same steps apply to any focused question.

Cochrane guidance is explicit that searches should be planned from eligibility criteria, use multiple sources, and combine free-text terms with subject headings. That planning stage is also where you define what counts as "one study" for your review. Before touching duplicates, write down:

• The unit of analysis. Is it the study, the trial registration, or the report? For synthesis you usually want one study with all its linked reports.
• What makes two records the same. Same trial registration number, same DOI, or same population/intervention/comparator with overlapping enrollment periods.
• What makes two records different. Different registration numbers, non-overlapping enrollment, or clearly different designs.

This definition is your tie-breaker later. Without it, every ambiguous pair becomes a judgment call made under time pressure.

Cochrane recommends using multiple sources and documenting the process. Practically, that means your raw export should carry a source label for every record — database name, search date, and the export batch. When you later find two near-identical records, the source label tells you whether you are looking at cross-database overlap or two reports of one trial.

If you are still assembling the candidate set, use a scholarly search interface to run and capture the searches, then keep the exports intact. WisPaper's scholarly search interface is one place to run academic paper searches; the key discipline is to save each search's output separately rather than merging everything into one undifferentiated file.

Checkpoint 1: Every record has a source, a search date, and a stable identifier field (DOI, PMID, or registration number) where one exists. If a large share of records lack identifiers, fix the export before deduplicating.

Start with the mechanical layer. Match on exact DOI, exact PMID, or exact trial registration number. These matches are safe to merge because they point to the same underlying object.

Then run a second pass on normalized bibliographic fields: author surname set, year, title after lowercasing and stripping punctuation, journal, and volume/pages. Flag — do not auto-merge — anything that matches on title but differs on identifier, or matches on authors and year but differs on title.

A reusable triage table keeps this honest:

Checkpoint 2: Automated merges are limited to exact identifier matches. Everything else is flagged for review.

For each flagged pair, compare in a fixed order so you do not anchor on the title alone:

1. Registration number. A shared registration number is strong evidence of one study.
2. DOI and PMID. Different identifiers with the same title usually mean versions, not duplicates.
3. Enrollment period and sample size. Overlapping periods and identical N suggest the same cohort; different periods suggest distinct studies.
4. Author list and affiliations. Useful but weaker than identifiers — large trials generate many papers from the same group.
5. Journal, volume, pages, year. Catches preprint-to-publication pairs and errata.

When a pair survives this comparison as "same study, different report," do not delete either record. Instead, create a study-level parent and attach both reports to it. That preserves the relationship the synthesis will need.

When you do merge true duplicates, keep a canonical record that retains:

• The most complete identifier set (DOI, PMID, registration number).
• The source labels from every merged copy, so the search documentation stays accurate.
• Any notes, screening decisions, or tags attached to the discarded copy.

Reference managers support this. Zotero's Duplicate Items collection is built for review and merging, and its collections let one item sit in multiple project groupings without being duplicated — useful when the same study belongs to both your pneumonia review and a broader anti-infective project. Tags can carry topic, method, status, or workflow labels, and saved searches update automatically as the library changes.

If you are organizing the retained set in a working library, WisPaper's My Library is where you can save or upload papers and organize them. Do the merge decisions first; file the cleaned set second.

Checkpoint 3: After merging, the count of unique studies plus linked reports reconciles with the raw record count minus confirmed duplicates. If it does not reconcile, you have an unexplained gap.

This is the step most often skipped. Cochrane guidance calls for linking multiple reports of one study, and PRISMA 2020 provides flow-diagram templates and reporting structures that make the search-to-selection path transparent. A study-level structure with linked reports serves both.

For the pneumonia example, one trial might yield:

• The primary results paper (mortality at 30 days).
• A secondary analysis of subgroup mortality.
• A long-term follow-up report.
• A protocol or registration entry.

All four belong to one study. Your synthesis extracts the 30-day mortality outcome from the report that actually contains it, while the study-level record prevents that trial from being counted four times.

Suppose your search returns:

• Record A: Trial of adjunctive therapy in community-acquired pneumonia, 30-day mortality reported, registration NCT-XXXX, DOI present.
• Record B: Same trial name, same registration number, different journal, reports a subgroup analysis.
• Record C: Similar title, different registration number, enrollment period two years later, different sample size.

Processing order:

1. A and B share a registration number → same study, two reports. Link them under one study record; keep both.
2. C differs on registration number, enrollment period, and sample size → distinct study. Keep separate.
3. If a fourth record D matched A on title but had a different DOI and an earlier year, treat it as a possible preprint or earlier version and check whether it reports the same enrollment.

Result: two studies, three reports. A naive title-based deduplication would have collapsed A, B, and possibly D into one record and lost the subgroup data; a naive identifier-only pass would have left A and B as two "studies" and double-counted the trial.

Once the study set is clean, the appraisal questions are distinct and should be recorded separately for each study:

• Evidence level — the design and what it can support (for example, a randomized trial versus an observational cohort).
• Applicability — whether the population, setting, and comparator match your review question.
• Indirectness — where the study deviates from your question (different comparator dose, different care setting, surrogate outcome).
• Study limitations — risk-of-bias concerns within the study itself.

Keeping these in separate fields prevents a common error: labeling a study "low quality" when the real issue is that it is indirect to your question. A well-conducted trial in a different setting is not a flawed trial; it is an indirect one. Record each dimension per study, not per report, then note which report supplies which outcome.

PRISMA 2020 supplies a checklist, expanded explanations, and flow-diagram templates for original and updated reviews. Use the diagram as an arithmetic check: records identified, duplicates removed, records screened, reports linked to studies, studies included. Cochrane's recommendation to document the search and selection process and to rerun searches before publication means your deduplication log should be dated and reproducible.

A minimal log entry per merge contains: the records merged, the matching field that justified the merge, the reviewer, and the date. For linked reports, log the study parent and the reason for linking.

• Eligibility criteria and the definition of "one study" are written down.
• Every raw record carries a source and search date.
• Exact-identifier duplicates are merged; all other matches are flagged.
• Every flagged pair was compared on registration number, DOI/PMID, enrollment, authors, and bibliographic fields.
• Multiple reports of one study are linked, not deleted.
• Merged records retain source labels and prior notes.
• Study-level appraisal separates evidence level, applicability, indirectness, and limitations.
• The deduplication log reconciles with the PRISMA flow counts.
• Searches are scheduled to be rerun before publication.

Stop and escalate to a second reviewer when:

• Two records share a title but have different registration numbers and overlapping enrollment periods.
• A record has no identifier and cannot be matched on any bibliographic field.
• Merging would discard the only report containing your primary outcome.
• The reconciliation in Checkpoint 3 does not balance and the gap cannot be traced to a logged merge.
• A study's applicability and indirectness judgments conflict between reviewers.

Do not resolve these by deleting the ambiguous record. Park it as "unresolved," keep it visible in the flow, and resolve it before synthesis.

For longer tasks — building the study-level structure, drafting the selection narrative, or checking a reference list — an agent workflow can carry the repetitive parts while you keep the merge decisions. WisPaper's Scholar Agent area covers literature review and related longer research tasks. If your concern is that a reference list contains entries that do not correspond to real papers, TrueCite checks BibTeX against academic databases to flag hallucinated references. Run that citation check after deduplication and use the identifier comparison above to resolve each flagged record.

Once the cleaned set is in your library, you can ask questions grounded in those papers through Library QA, which works on papers uploaded to My Library or added from search results. That is most useful after the study-report structure is settled, so answers map to studies rather than to duplicate records.