Removing duplicate records before screening is essential to avoid wasting time reviewing the same paper multiple times and to prevent double-counting in your systematic review.
Why Duplicates Occur
When you search multiple databases (PubMed, Scopus, Web of Science, etc.), the same paper often appears in several results because databases have overlapping coverage. Different databases may also record the same paper with slight variations in title formatting, author names, or metadata, making duplicates harder to spot manually.
Manual Deduplication Steps
1. Export and Consolidate
Export all search results from each database into a single reference manager (Zotero, EndNote, Mendeley, or WisPaper's My Library, which automatically extracts metadata and organizes references in one place).
2. Use Built-in Duplicate Detection
Most reference managers offer automatic duplicate detection based on DOI, title, and author matching. Run this function first—it catches exact matches immediately.
3. Review Flagged Matches Manually
Automated tools may miss near-duplicates (e.g., "COVID-19" vs. "SARS-CoV-2" in titles, different publication years for preprints vs. final versions). Manually review flagged pairs and check DOIs when available—DOIs are the most reliable unique identifier.
4. Standardize Metadata
Fix inconsistent author names (e.g., "Smith, J." vs. "Smith, John") and title capitalization before running duplicate checks again.
When to Remove Duplicates
Remove duplicates immediately after importing and before title/abstract screening. This ensures your PRISMA flowchart accurately reflects the number of unique records screened. Document how many duplicates you removed—this transparency is required in systematic review reporting.
Common Pitfall
Don't delete duplicates blindly. Sometimes what looks like a duplicate is actually a related paper (e.g., a protocol paper and the final trial results). Always verify before removing.
By cleaning your dataset upfront, you'll save hours during screening and ensure your final analysis is based on unique, verified sources.
