The Evidence Looks Strong but the Patients Are Different: A Clinical Applicability Case

A guideline panel is drafting a recommendation for a common chronic condition. The lead trial is large, randomized, and widely cited. The effect on the primary outcome is statistically significant and clinically meaningful. Then a committee member asks the question that stalls the meeting: "Our clinic population is older, has more comorbidity, and takes more concurrent medications than the trial cohort. Does this evidence apply to them?"

The honest answer is not yes or no. It is a structured judgment about evidence level, applicability, indirectness, and study limitations — four things that are often collapsed into a single sentence like "the evidence is strong." This article walks through a worked case, shows where the first draft of the evidence summary went wrong, and shows the corrected deliverable. The result is a defensible applicability appraisal for research, guideline development, or scholarly use.

To keep the reasoning concrete, fix the question tightly. Suppose the panel is appraising a randomized trial of a medication for a chronic condition, with a primary outcome of a composite clinical event over a fixed follow-up period. The trial enrolled adults meeting explicit eligibility criteria; the panel's target population is adults seen in a general clinic, including people with multimorbidity and polypharmacy.

A focused question has five parts: population, intervention, comparator, outcome, and setting. The panel's question is narrower than "does this drug work?" It is: in our clinic population, compared with usual care, does this intervention reduce the composite event outcome, and how much of the trial's result survives the differences between the trial cohort and our patients?

That framing matters because applicability is judged against a specific population and outcome, not in the abstract. A trial can be high quality and still be only partially applicable to a given clinic.

When a summary says the evidence is strong, it usually means the study design was rigorous and the result was precise. That is a statement about internal validity. It does not automatically answer whether the result transfers to a different population, setting, or baseline risk.

Four judgments need to be kept separate:

• Evidence level — how the study was designed and how directly it addresses the question (for example, a randomized trial versus an observational study).
• Applicability — how closely the trial's population, setting, intervention delivery, and outcome match the panel's target context.
• Indirectness — whether the evidence addresses a different population, intervention, comparator, or outcome than the question asks about.
• Study limitations — risks of bias within the study itself, such as problems with randomization, blinding, attrition, or outcome ascertainment.

A trial can have a high evidence level and low applicability. It can also be directly applicable but carry important limitations. Collapsing these into one label hides exactly the information the panel needs.

The first draft of the evidence summary was built from a single search string typed into a database and a handful of familiar citations. That is a fragile starting point. Cochrane guidance on searching recommends planning the search from the eligibility criteria, using multiple sources, and combining free-text terms with subject headings.

For this case, the eligibility criteria define the population (adults with the chronic condition, including multimorbidity), the intervention, the comparator, and the outcome. Each of those becomes a search concept. The search should also capture the setting where it matters — for example, general clinic versus specialist referral populations.

A practical repair is to draft the concepts first, then translate them into database syntax. That is where a scholarly search interface helps: you can run the concept blocks and inspect what comes back before committing to a screening set. WisPaper's scholarly search interface is built for finding academic papers, which fits this stage of building and testing the search.

Free-text searching alone will miss records that describe the same concept with different words. NLM's Medical Subject Headings provide a controlled and hierarchical vocabulary: preferred headings, entry terms, scope notes, qualifiers, and tree positions can be used to expand or focus a PubMed strategy.

In practice, that means checking whether the population concept has a preferred heading, whether narrower terms under it should be included, and whether a qualifier changes the meaning. It also means reading the scope note rather than assuming a heading means what it sounds like.

PubMed's own documentation notes that search details reveal how terms were translated and combined. That is a useful audit step: after running a search, inspect the translation to see whether your terms were mapped as intended or silently altered.

Cochrane guidance also recommends documenting the search process, deduplicating records, screening in stages, and linking multiple reports of one study. This is not administrative overhead. In an applicability appraisal, the same trial may appear as a primary report, a subgroup analysis, and a long-term follow-up. If those are treated as three separate studies, the evidence base looks larger and more consistent than it is.

A staged approach helps:

1. Record the search date, sources, and exact strings.
2. Deduplicate before screening.
3. Screen titles and abstracts, then full texts.
4. Group reports by study before extracting results.
5. Rerun the search before finalizing, as Cochrane recommends.

When the panel needs to organize the included reports and keep the primary trial, subgroup paper, and follow-up together, a library workspace is the natural place. WisPaper's My Library lets you save or upload papers and organize them, which supports keeping linked reports of one study in one place.

With the study set assembled, the panel can now make the four judgments separately. A simple table forces the discipline:

Filling this table for the case produces a more useful summary than "strong evidence." The trial may be a randomized design (high evidence level) while the population is younger and healthier than the clinic (applicability concern), and the outcome may be a composite that weights components differently than the panel cares about (indirectness in the outcome).

The most common applicability error is to take a relative effect from the trial and apply it to a population with a different baseline risk. If the clinic population has a higher baseline risk of the outcome, the same relative effect produces a larger absolute benefit; if lower, a smaller one. The relative effect alone does not tell the panel what to expect.

So the appraisal should record the trial's baseline event rate alongside the panel's best estimate for its own population, and state the absolute difference implied by each. Where the panel's baseline risk is uncertain, say so and show the range. That is more honest than importing the trial's absolute numbers wholesale.

A composite outcome can hide disagreement. If the trial's composite is driven mainly by one component, and the panel cares most about another, the result may be less informative than it appears. The appraisal should record which components drove the trial result and whether the panel's priority outcome is represented.

Similarly, the intervention as delivered in the trial — dose, monitoring, follow-up intensity — may differ from what the clinic can deliver. That is an applicability issue, not a study limitation, and it belongs in its own row.

Once the included papers are organized, the panel often has specific questions that require reading across them: what were the exclusion criteria, what was the baseline risk in the control arm, how was the composite defined, and which components drove the result. Answering these by re-reading every PDF is slow.

A question-answering step over the uploaded papers can speed this up. WisPaper's Library QA supports asking questions based on papers uploaded to My Library or added from search results. Used after the study set is assembled, it helps the panel locate the specific details that feed the applicability table.

Applicability appraisals are citation-dense. A draft may cite the primary trial, a subgroup analysis, a follow-up report, a methods paper, and several background references. A mistyped or fabricated reference in that list undermines the whole document.

A citation-screening step catches this before circulation. WisPaper's TrueCite checks BibTeX against real academic databases to flag hallucinated references. Running the reference list through that check is a concrete final step before the deliverable goes to the panel.

The repaired evidence summary looks different from the first draft. Instead of "strong evidence supports the intervention," it states, in order:

1. The question, with population, intervention, comparator, outcome, and setting named explicitly.
2. The evidence level, with the design stated and the reason it addresses the question.
3. Applicability, with the trial population compared to the clinic population on age, comorbidity, polypharmacy, and baseline risk.
4. Indirectness, with any mismatch in population, intervention delivery, comparator, or outcome definition named.
5. Study limitations, listed separately from applicability concerns.
6. Absolute effects, shown for the trial's baseline risk and for the panel's estimated baseline risk, with uncertainty stated.
7. A conclusion that says how much of the trial result is expected to transfer, and what remains uncertain.

That structure lets a reader see which judgment drives the conclusion. It also makes the appraisal updatable: if a new subgroup report appears, it slots into the relevant row rather than forcing a rewrite.

Several lessons from this case apply to any applicability appraisal:

• Fix the question before searching. Population, intervention, comparator, outcome, and setting should be written down first; the search follows from them.
• Keep the four judgments separate. Evidence level, applicability, indirectness, and study limitations answer different questions and should not be merged into one label.
• Group reports by study. Multiple reports of one trial are not multiple trials.
• Compare baseline risk explicitly. Relative effects do not transfer unchanged across populations with different baseline risk.
• Check the outcome definition. A composite that weights components differently than the panel's priority outcome is an indirectness issue.
• Document and rerun. A recorded search that is rerun before finalizing is more defensible than a one-time query.

For longer appraisals that span many papers, a structured research workspace can hold the question, the included studies, and the extracted details together. WisPaper's Research Projects area supports creating a project for that kind of work.