Reviewers should assess whether the research question is important, the methodology is sound and feasible, and the analysis plan is appropriate—before any data collection begins.
Focus on Pre-Data Quality, Not Results
In a Registered Report, Stage 1 review centers on evaluating the theoretical rationale and methodological rigor before results exist. Reviewers should judge whether the proposed study, if conducted as described, would produce meaningful and interpretable findings—regardless of outcome. This shifts evaluation away from novelty of results toward quality of design.
Key Evaluation Criteria
Research Question and Theory: Is the question clearly defined and theoretically motivated? Does the introduction justify why this study matters and what gap it addresses? Reviewers should assess whether the hypotheses are specific and testable.
Methodology and Design: Is the proposed method appropriate for answering the research question? Check sample size justifications (power analyses), inclusion/exclusion criteria, measurement instruments, and procedural details. The design should be sufficiently detailed that another team could replicate it. When evaluating complex multi-study protocols, WisPaper's Scholar QA can help reviewers quickly verify methodological claims against cited literature, with every answer traceable back to the exact source paragraph.
Analysis Plan: Are the statistical or analytical approaches pre-specified and appropriate? Reviewers should evaluate whether the analysis plan matches the hypotheses, includes contingencies for different data scenarios, and addresses potential confounds. Look for clarity on how results will be interpreted under different outcomes.
Feasibility: Can the authors realistically complete this study? Consider resource availability, ethical approval status, recruitment plans, and timeline.
What Not to Evaluate
Do not reject based on predicted results or perceived likelihood of "positive" findings. Avoid requiring authors to defend why negative results would be publishable—that's the point of pre-registration. Novelty of expected outcomes should not influence acceptance.
Provide Constructive Feedback
Suggest specific improvements to design, measures, or analysis rather than vague concerns. If a protocol has fixable flaws, request revisions rather than outright rejection. The goal is to help authors strengthen their study before data collection, when changes are still possible.
