Reading a preregistered study critically means checking whether the researchers actually followed their own plan, not just trusting the registration stamp. A preregistration is a time-stamped, public commitment to hypotheses, methods, and analyses made before data collection — but it only protects against bias if you verify the execution.
Step 1: Locate the preregistration and compare it to the paper
Start by finding the preregistration (often on OSF, AsPredicted, or ClinicalTrials.gov) and open the published paper side by side. Look for the registration ID in the methods section or footnotes. If you can’t find one, that’s a red flag — the study may be “pre-registered” in name only.
Step 2: Check for three common deviations
- Hypothesis switching – Did the primary outcome in the paper match the one registered? If the authors swapped the main outcome for a secondary one (often because the first wasn’t significant), the study’s conclusions lose their confirmatory power.
- Analysis plan drift – Compare the statistical tests, covariates, and exclusion criteria. Minor wording changes are fine, but adding post-hoc subgroups or switching from a preregistered Bayesian to frequentist test is a red flag.
- Sample size changes – Was the stopping rule (e.g., “collect until N=200”) met? If they stopped early or added participants after peeking at results, the p-values are no longer trustworthy.
Step 3: Distinguish confirmatory from exploratory findings
A well-done preregistration will clearly label which analyses were planned (confirmatory) and which were discovered after seeing the data (exploratory). When reading, ask yourself: Are the headline claims based on the confirmatory tests? If the authors hang their conclusions on exploratory analyses, treat those as hypotheses to be tested later, not as established facts.
Step 4: Evaluate the preregistration’s quality itself
Not all preregistrations are equal. A vague one (e.g., “we will examine relationships between variables”) provides little protection. Look for specific, falsifiable predictions, detailed operationalizations, and pre-specified effect sizes. If the registration is too generic, the authors have room to rationalize almost any result.
Step 5: Use tools that flag deviations automatically
Manually cross-checking every number is tedious and error-prone. If you’re reviewing several studies, consider using WisPaper’s Scholar QA — it can pull the preregistration and the paper side-by-side and trace claims back to exact pages or paragraphs, making it easier to spot whether the reported methods match the registered plan.
Finally, remember that preregistration is a tool, not a guarantee. A study can be perfectly preregistered and still be poorly designed or misinterpreted. Your job as a critical reader is to verify the chain from plan to execution — and to be especially skeptical when the registered plan and the published story diverge without a clear, honest explanation.
