Extracting methods consistently means turning a stack of papers into one comparable record: the same fields, the same definitions, and the same decision rules applied to every study. The reliable way to get there is to treat method extraction as a small, documented process rather than a reading habit. You define what counts as a method before you read, build a fixed template, pilot it on a few papers, then apply it paper by paper with a log of your judgment calls.
This guide is written for students and early-career researchers finishing a course paper, thesis chapter, or first manuscript and preparing the next project. It walks through the starting state, the decisions and artifacts you need, a complete sequence with checkpoints, a worked example, a reusable checklist, and clear completion criteria.
Why method extraction drifts, and what consistency actually requires
Method extraction drifts for predictable reasons. Papers report methods at different levels of detail, use different names for similar procedures, and bury key steps in supplementary material or in cited prior work. When you read without a fixed frame, you notice different things in each paper, and your notes become a set of summaries rather than a comparable dataset.
Consistency, then, is not about reading more carefully. It requires three things working together:
- A defined scope. You decide in advance which studies are eligible and which method details you will record.
- A fixed template. Every paper is read against the same fields, in the same order.
- A decision log. When a paper is ambiguous, you record the rule you applied so the next paper is handled the same way.
The search and screening literature reinforces this discipline. The Cochrane Handbook recommends planning searches from eligibility criteria, using multiple sources, combining free-text terms with subject headings, documenting the process, deduplicating records, screening in stages, and linking multiple reports of one study. Those habits apply directly to method extraction: eligibility criteria define what you extract from, staged screening keeps your set clean, and linking multiple reports prevents you from treating one study as several.
Diagnose your starting state before you build anything
Most people arrive at this task with one of a few recognizable starting states. Naming yours tells you which step to fix first.
The pile of PDFs. You have collected papers but have no eligibility rule, so the set keeps growing and shrinking. Fix: write inclusion criteria before reading further.
The inconsistent notes. You have summaries, but each one emphasizes different details. Fix: build the template and re-extract a small pilot set.
The supervisor's comments. Feedback says your method comparison is uneven or that you describe studies without comparing them. Fix: define the fields your comparison needs, then re-read with those fields in view.
The fresh project. You are starting a new project, perhaps in January, and want a method baseline before designing anything. Fix: run the full sequence below on a deliberately bounded set.
A winter-break cleanup or a January project start is a good moment to do this properly, because you can invest in a reusable template that serves the next manuscript too.
Decide what counts as a method before you read
This is the decision that determines everything downstream. Write down, in plain language, what you will and will not record.
Ask yourself:
- Which studies are eligible? Define the population, intervention or exposure, comparison, and outcomes that matter to your question, and write them as explicit criteria.
- What counts as a method detail? Decide whether you record study design, sample and sampling, materials and instruments, procedure, analysis approach, and reported limitations. Choose the fields your own comparison actually needs.
- How deep will you go? Decide whether you follow cited prior-work methods or record only what the paper itself reports. State the rule.
- What will you do with missing detail? Decide whether you mark it as not reported, follow the citation, or exclude the study. State the rule.
Write these decisions into a short protocol document. It does not need to be long, but it must exist before you start extracting, so that later papers are judged against the same standard as earlier ones.
Build a fixed extraction template
A template is what makes extraction comparable. Keep it narrow enough to complete for every paper and specific enough to support your comparison.
A workable template records, for each study:
- Citation and identifier — authors, year, title, and a stable ID you assign.
- Eligibility check — whether the study meets your criteria, and why.
- Study design — the design as the authors describe it.
- Sample — who or what was studied, and how they were selected.
- Materials and instruments — what was used to collect data.
- Procedure — the steps the authors report, in order.
- Analysis approach — how the authors say they analyzed the data.
- Reported limitations — what the authors themselves flag.
- Extraction notes — ambiguities, judgment calls, and where in the paper each detail came from.
Keep the fields identical across papers. If you find yourself adding a field mid-stream, add it to the template and go back to fill it for earlier papers rather than applying it only to new ones.
Run a pilot on a small set and check your own agreement
The pilot is where consistency is won or lost. Take three to five papers that clearly meet your criteria and extract them using the template. Then compare your extractions against each other.
Look for:
- Fields you filled differently for similar details.
- Fields you left empty because the paper was unclear rather than because the detail was absent.
- Judgment calls you made without recording them.
Revise the template and the protocol to resolve each discrepancy, then re-extract the pilot set. When two passes over the same papers produce the same records, your template is ready to scale.
This is also the point to bring in tools that support the reading task. If you are working from a set of papers you have saved, you can organize them in My Library so that the pilot set and the full set stay separate and identifiable. If you want to ask questions grounded in those saved papers, Library QA lets you query papers uploaded to My Library or added from search results, which is useful for checking whether a detail you expected is actually reported.
Find and screen the papers you will extract from
Consistent extraction depends on a clean, bounded set. Build that set deliberately.
Search from your criteria. Turn your eligibility criteria into search terms. Combine free-text terms with subject headings where the database supports them, and use multiple sources rather than relying on one. PubMed, for example, supports field tags, phrase searching, filters, Advanced Search, saved searches, and Similar Articles, and its search details reveal how terms were translated and combined. Google Scholar documents searching by author and title, date sorting and limits, Cited by and Related articles, citation export, and alerts. Use these features to make your search reproducible rather than ad hoc.
Document the search. Record the sources, the terms, the date, and the filters. Cochrane's guidance to document the process and deduplicate records applies here directly.
Screen in stages. Screen titles and abstracts first, then full texts, applying your eligibility criteria at each stage. Link multiple reports of the same study so one study does not enter your set twice.
Rerun before you finalize. Cochrane recommends rerunning searches before publication. If your extraction feeds a manuscript, a final search pass catches studies published while you were working.
For finding papers through WisPaper's scholarly search interface, start at WisPaper search. If you prefer a public entry point that explains the search capability, use WisPaper Scholar Search.
Extract paper by paper with a decision log
With a clean set and a tested template, extraction becomes a repeatable routine. Work through papers one at a time, in a fixed order, and complete every field before moving on.
For each paper:
- Confirm eligibility against your written criteria and record the decision.
- Read for method, not for results. Locate the design, sample, materials, procedure, and analysis sections, plus any supplementary material.
- Fill every template field, marking "not reported" where the paper is silent rather than guessing.
- Record the location of each detail so you can verify it later.
- Log judgment calls in the extraction notes, with the rule you applied.
- Update the protocol if a new situation arises, then revisit earlier papers to apply the new rule.
When a paper is long or methodologically dense, a longer research task can help you work through the literature systematically. WisPaper's Scholar Agent area is designed for longer research tasks with literature review, inspiration, or reproduction use cases. Use it to support the reading work, then record the extracted details in your own template so the record stays comparable across all papers.
Worked example: a January project start
Suppose you are starting a new project in January and need a method baseline from the literature before you design your own study.
Step 1 — Scope. You write eligibility criteria: studies that address your research question, published within a defined window, reporting an empirical method. You decide to record design, sample, materials, procedure, analysis, and reported limitations, and to mark unreported details rather than infer them.
Step 2 — Search. You translate the criteria into terms, search more than one source, and save the search details. You record sources, terms, date, and filters, and deduplicate the records.
Step 3 — Screen. You screen titles and abstracts, then full texts, applying the criteria at each stage. You link multiple reports of the same study so each study appears once.
Step 4 — Pilot. You extract three clearly eligible papers, compare the records, and find that you recorded "sample" inconsistently: sometimes the number of participants, sometimes the recruitment setting. You revise the field into two subfields, re-extract the pilot, and confirm the records now match.
Step 5 — Extract. You work through the full set in a fixed order, filling every field and logging judgment calls. When one paper cites a prior work for its procedure, you apply your pre-written rule and mark the procedure as reported by citation.
Step 6 — Organize. You keep the project's papers and notes together in Research Projects, so the extraction record sits alongside the rest of the project rather than in scattered files.
Step 7 — Verify citations. Before you write up, you check that every reference in your extraction record is real. TrueCite screens BibTeX citations against real academic databases to flag hallucinated references, which is a useful final check on the bibliography your template depends on.
Step 8 — Rerun and finalize. You rerun the search before finalizing, add any newly eligible studies, extract them with the same template, and close the set.
A reusable checklist
Use this checklist for each extraction project. It compresses the sequence above into a form you can reuse.
| Stage | Checkpoint | Artifact |
|---|---|---|
| Scope | Eligibility criteria written and dated | Protocol document |
| Template | Fields fixed and defined | Extraction template |
| Search | Sources, terms, date, filters recorded | Search log |
| Screening | Staged screening complete; duplicate reports linked | Screened set with decisions |
| Pilot | Two passes agree on the pilot set | Revised template and protocol |
| Extraction | Every field filled; judgment calls logged | Completed extraction records |
| Verification | Citations checked against databases | Verified bibliography |
| Finalization | Search rerun; new studies extracted | Closed, final set |
Keep the protocol, template, search log, and extraction records together. They are the evidence that your extraction was consistent, and they make the next project faster.
How to know your extraction is ready
Consistency is a property you can check, not a feeling. Your extraction is ready when all of the following hold:
- The protocol predates the extraction. Your eligibility criteria and field definitions were written before you extracted, and any later changes are dated and applied to earlier papers.
- Every paper was read against the same template. No paper has fields that others lack, and no field was added only for later papers.
- The pilot agreed with itself. Two passes over the pilot set produced the same records after revision.
- Judgment calls are logged. Every ambiguity has a recorded rule, and the rule was applied consistently.
- Missing detail is marked, not guessed. Fields the paper does not report are marked as not reported under your stated rule.
- The set is clean and closed. Duplicate reports are linked, the search is documented, and a final rerun has been completed.
- Citations are verified. The references your extraction depends on have been checked.
If any of these fail, return to the relevant stage rather than patching individual records. Consistency comes from the process, not from correcting outputs one at a time.
Your next action
Pick the smallest version of this workflow and run it this week. Write your eligibility criteria and template on one page, choose three papers that clearly meet the criteria, extract them twice, and compare. Revise the template until the two passes agree. That pilot is the foundation for the rest of your set, and it is the artifact you can show a supervisor when they ask how your method comparison was built.
If you are starting fresh and want a general entry point to the tools described here, begin at the WisPaper homepage or the WisPaper English homepage.




