Source evaluation is not a single judgment about whether a paper is "good." It is a chain of decisions: what counts as eligible evidence for your question, how you found it, whether you can trace it, how much weight it can carry, and whether the record you hand to someone else is complete enough to be checked. Teams producing client decks, decision memos, handoffs, and reusable evidence bases need that chain to be visible, because the reader of the final artifact is often not the person who did the searching.
This guide treats evaluation as a workflow with artifacts at each stage. It draws on documented practice from the Cochrane Handbook's guidance on searching and selecting studies, the PRISMA 2020 reporting standard, and the search mechanics documented by Google Scholar and PubMed. It also shows where WisPaper tools fit into specific steps.
What does "evaluating a source" actually require?
Most weak evaluations fail because they answer the wrong question. "Is this a reputable journal?" and "is this paper relevant?" are both useful, but neither tells you whether the source can support the claim you are making.
A working definition: a source is adequately evaluated when you can state (1) why it was eligible for your question, (2) how you found it and whether that search was systematic enough for your purpose, (3) what specific claim it supports and at what confidence, and (4) where the record lives so a colleague can retrieve and re-check it.
That definition implies four evaluation layers, each with its own artifact:
| Layer | Question it answers | Artifact you produce |
|---|---|---|
| Eligibility | Should this source be in scope at all? | Written inclusion/exclusion criteria |
| Discovery | Did I search in a way I can describe and repeat? | Search log with sources, terms, dates, filters |
| Appraisal | What can this source support, and how strongly? | Claim-to-source map with confidence levels |
| Provenance | Can someone else trace and reuse this? | Library record, citation file, project handoff note |
Skipping a layer is the most common cause of rework. A deck built on sources with no search log cannot be defended when a client asks how the evidence was selected; a memo with strong claims but no claim-to-source map cannot be updated when new evidence arrives.
Where is your team starting from?
Different readers enter this workflow at different points. Diagnose your starting state before choosing tools or templates.
Starting state A: a question but no source set. You have a research question or a client ask, and nothing collected. Start with eligibility criteria, then discovery. Do not begin by reading whatever surfaces first.
Starting state B: a pile of PDFs and links. Someone has already gathered material, but there is no record of how it was found or why it was kept. Start with a retroactive eligibility pass: for each item, write one line on why it is in scope. Items you cannot justify become candidates for removal or for a documented "background only" label.
Starting state C: a draft with citations. The argument exists and references are attached, but nobody has checked whether each citation supports the sentence it sits next to. Start with the claim-to-source map and work backward into discovery only where gaps appear.
Starting state D: a handoff or closeout. The work is done and must transfer to another team or a future project. Start with provenance: confirm every retained source has a retrievable record, and that confidence levels are attached to the conclusions, not buried in someone's memory.
Naming your starting state prevents the classic failure of running a full systematic search when a quick scoping answer would do, or of shipping a deck built on an unsystematic pile.
How do you set eligibility criteria before searching?
The Cochrane Handbook's guidance on searching for and selecting studies begins with planning searches from eligibility criteria. That ordering matters: criteria written before searching protect you from rationalizing whatever you happen to find.
For academic writing and enterprise research alike, write criteria as a short table with four rows:
- Population or subject: what kind of study, organization, dataset, or case is in scope.
- Intervention or phenomenon: the specific practice, technology, or relationship you are examining.
- Outcomes or claims of interest: what the source must actually report to be useful.
- Design and date limits: acceptable study types, publication windows, language, and any source types you will exclude.
Keep the criteria narrow enough to exclude something. A criterion that every paper passes is not doing work. If your question is exploratory, say so and define a broader but still explicit scope, such as "peer-reviewed empirical studies plus authoritative guidance documents from the last five years."
Store the criteria with the project, not in a personal note. When the work moves to a colleague, the criteria are the first thing they need in order to judge whether the source set is still valid.
How should you search so the source set is defensible?
Search quality is a property of the process, not of any single result. The Cochrane Handbook recommends using multiple sources, combining free-text terms with subject headings, documenting the process, deduplicating records, screening in stages, linking multiple reports of one study, and rerunning searches before publication. Not every project needs the full systematic-review apparatus, but the underlying logic scales down.
Use more than one entry point. A single database or a single search engine produces a biased slice. PubMed documents field tags, phrase searching, filters, Advanced Search, saved searches, and Similar Articles, and its search details show how terms were translated and combined. Google Scholar documents author and title searching, date sorting and limits, Cited by and Related articles, citation export, and locating alternative versions of a work. These mechanics are useful for both finding and for auditing what a search actually did.
Log as you go. For each search, record the platform, the exact terms, filters applied, date run, and number of records retrieved. This log is what lets you describe your method in a methods section or a client appendix without reconstructing it from memory.
Deduplicate before screening. The same study often appears as a preprint, a conference paper, and a journal article. The Cochrane Handbook's guidance on linking multiple reports of one study is directly relevant: decide at the criteria stage whether linked reports count as one source or several, and record the decision.
Screen in stages. Title and abstract screening first, then full-text review against the same criteria. Record reasons for exclusion at full-text stage; those reasons are often the most persuasive part of a methods appendix.
When you need to build the initial candidate pool, WisPaper search provides a scholarly search interface for finding academic papers, and the Scholar Search entry point explains the paper-finding capability for readers new to the tool. Use whichever discovery route you choose consistently, and log it the same way you would log a PubMed or Google Scholar query.
How do you appraise a source against a specific claim?
Appraisal is where most evaluation effort should go, and where generic checklists help least. The productive unit is the pair: one claim, one source.
For each claim in your draft or deck, record:
- The claim as written, in the exact wording that will appear in the artifact.
- The supporting passage, quoted or closely paraphrased with a page or section locator.
- The source's own scope: what population, setting, or dataset the finding comes from.
- A confidence level with a one-line reason.
A three-level confidence scheme is usually enough for enterprise artifacts:
| Level | Meaning | Typical basis |
|---|---|---|
| High | The claim is stated directly in a source whose scope matches yours | Direct finding in a comparable population or setting |
| Moderate | The claim is supported but requires an inferential step | Adjacent population, indirect outcome, or single source among several |
| Low | The claim is suggestive only | Exploratory finding, narrow case, or a source whose scope differs from yours |
Two appraisal habits prevent the most common errors. First, separate what the source found from what you want to conclude; a finding in one setting does not automatically transfer. Second, check whether multiple cited sources are actually independent. Several reports of one study are one piece of evidence, not several.
For teams working through a longer literature review, Scholar Agent supports longer research tasks in the literature review, inspiration, and reproduction use cases. Use it to accelerate the appraisal pass, then attach your own confidence levels to each claim; the confidence judgment belongs to your artifact, not to the tool.
What does a traceable evidence record look like?
A source is only as reusable as its record. At minimum, each retained source should have:
- A stable identifier (DOI or equivalent) and full citation.
- The search or referral path by which it entered the set.
- Its eligibility justification, in one line.
- The claims it supports, with confidence levels.
- A local copy or link that team members can open.
This is where a shared library earns its place. My Library lets you save or upload papers and organize them in the WisPaper library, which turns scattered PDFs into a single organized set that a colleague can inherit. Pair the library with a claim-to-source map kept in the project workspace so the two records stay in sync.
For questions that require interrogating the collected set rather than searching afresh, Library QA lets you ask questions based on papers uploaded to My Library or added from search results. This is useful at the appraisal stage, when you want to check whether a claim is actually present across your retained sources before assigning it a confidence level.
How do you verify citations before a handoff?
Citation errors cluster at the end of projects, when references are assembled quickly from notes, drafts, and earlier decks. Two failure modes dominate: references that do not correspond to a real publication, and references that exist but do not support the sentence they are attached to.
The second failure is an appraisal problem and is caught by the claim-to-source map. The first is a record-integrity problem and can be screened mechanically. TrueCite checks BibTeX against real academic databases to flag hallucinated references, which makes it a sensible gate before a bibliography is frozen into a deliverable.
Run the check as a defined step, not an afterthought: export the BibTeX, screen it, resolve every flag, and only then attach the bibliography to the deck or memo. Record that the screen happened, along with the date, in the project log.
How do you report the method so readers can trust the result?
PRISMA 2020 provides a reporting checklist, expanded explanations, an abstract checklist, and flow-diagram templates for original and updated systematic reviews, including transparent reporting of search, selection, and synthesis methods. Even when your work is not a systematic review, the PRISMA structure is a useful model for what a methods appendix should contain: what was searched, how records were selected, and how evidence was synthesized.
For enterprise artifacts, a one-page method note covering the same elements is usually sufficient:
- Eligibility criteria, stated in full.
- Sources searched and dates, with exact terms for at least the primary searches.
- Counts at each screening stage, with reasons for full-text exclusions.
- How linked reports were handled.
- How confidence levels were assigned.
If your project is a formal review intended for publication, follow the PRISMA 2020 checklist directly rather than the abbreviated version. If it is a decision memo, the one-page note still does the essential job: it lets a skeptical reader see how the evidence was assembled.
Worked example: a decision memo with provenance and confidence
Consider a team asked to recommend whether to adopt a new operational practice. The deliverable is a two-page decision memo for an internal steering group, plus a handoff pack for the team that will implement or revisit the decision.
Step 1: Criteria. The team writes scope criteria: empirical studies or authoritative guidance published in the last five years, in settings comparable to their own operation, reporting outcomes relevant to the decision. Studies from unrelated sectors are marked background only.
Step 2: Search log. Two discovery routes are used, with terms, filters, and dates recorded. Records are deduplicated, and linked reports of the same study are grouped.
Step 3: Screening. Title and abstract screening removes clearly out-of-scope records; full-text review removes the rest, with a one-line reason for each exclusion.
Step 4: Claim-to-source map. Each memo claim gets a row: the claim, the supporting passage, the source's scope, and a confidence level. The memo's central recommendation is marked Moderate because the strongest supporting findings come from adjacent settings; a secondary claim is marked High because a directly comparable study reports it.
Step 5: Citation screen. The bibliography is exported as BibTeX and screened before the memo is circulated.
Step 6: Handoff. The retained sources live in a shared library; the criteria, search log, screening counts, claim map, and confidence rationale live in the project workspace. The handoff note states which claims would need re-examination if new evidence appeared.
Research Projects provides the WisPaper Research Projects area where a team can create a project to hold this work together, so the memo and its evidence trail travel as one package rather than as a document plus a folder of loose PDFs.
How do you know the evaluation is ready to ship?
Readiness is checkable. Before a deck, memo, or handoff leaves the team, confirm each of the following:
- Every claim in the artifact appears in the claim-to-source map with a confidence level.
- Every retained source has a stable identifier and a retrievable record.
- The search log is complete enough that a colleague could rerun the primary searches.
- Screening counts and exclusion reasons are recorded.
- Linked reports of the same study are grouped, and the grouping decision is documented.
- The bibliography has passed a citation screen.
- The method note is attached or linked.
If any item is missing, the artifact is not unready in some vague sense; it is missing a specific, fixable component. Treat the list as a gate with named owners rather than a general quality aspiration.




