You open the matrix a week before the manuscript goes back to coauthors. Every row has a paragraph of notes. Some are two sentences, some are a page. One says "strong methods, small sample." Another says "relevant but dated." A third is a block quote with no label. When a coauthor asks which studies actually measured the outcome you care about, you cannot answer from the matrix. You can only re-read the papers.
That is the failure this guide addresses. The matrix is not empty; it is unstructured. Summaries record what a reader thought was interesting, but they do not record the same attributes for every study, so nothing can be compared, counted, or audited. The repair is not more writing. It is deciding, in advance, which fields every row must carry, then rebuilding the rows against those fields and preserving a trail that shows how each value was reached.
This article walks through diagnosis, the decisions that fix the structure, the artifacts you need, and the test for whether the rebuilt matrix is ready to support a manuscript, a handoff, or a review.
Why do summary-only matrices fail at the comparison stage?
A summary is written for the person who wrote it. A comparable field is written for the question the project is trying to answer. The difference shows up the moment someone else needs the matrix.
Summary-only matrices fail in three predictable ways:
- Uneven granularity. One row captures sample size and design; the next captures only a theme. You cannot tell whether a blank means "not reported" or "not checked."
- Embedded judgment. Phrases like "strong" or "dated" hide the criterion behind them. A coauthor cannot agree or disagree with a judgment that has no stated basis.
- No link back to the source. A summary that does not say where in the paper a claim came from cannot be verified later, during peer review or a lab handoff.
None of these are writing problems. They are design problems. The fix is to define the columns before filling them, and to treat each cell as a claim that needs a location in the source.
Diagnose your starting state before you rebuild
Before adding fields, characterize what you actually have. A short diagnostic saves you from rebuilding rows that were fine and from trusting rows that were not.
Check whether your rows are studies or reports
Cochrane's guidance on searching and selecting studies notes that a single study can have multiple reports. If your matrix has one row per PDF, two rows may describe the same study, and your counts will be wrong. Decide whether the unit of a row is a study or a report, and record that decision. If it is a study, you need a field that links multiple reports to one record.
Check whether blanks are meaningful
Sort each column and look at the empty cells. For every blank, ask: did we check and find nothing, or did we never look? Those are different states and should be different values. A matrix that cannot distinguish them will mislead whoever reads it next.
Check whether your inclusion logic is written down
Cochrane recommends planning searches from eligibility criteria. If your eligibility criteria exist only in a coauthor's head, your rows cannot be checked against them. Write the criteria down first; the fields follow from them.
Decide the comparable fields from your eligibility criteria
Fields should not be a generic template. They should be the attributes your question requires you to compare. Derive them from the eligibility criteria and from the synthesis you plan to run.
A workable set usually includes:
- Identity: study label, linked reports, and the source location for each extracted value.
- Eligibility: the criterion-by-criterion decision, with the reason for exclusion recorded rather than deleted.
- Design and population: the attributes your comparison depends on, each with a defined value set.
- Outcome data: the measure, the time point, and where in the report it appears.
- Certainty notes: what the report itself flags as a limitation, tied to a location.
Two rules keep the fields comparable. First, every field gets a defined value set or a stated format, so two coders produce the same kind of entry. Second, every extracted value carries a pointer to its location in the source. A value without a location is a summary in disguise.
Build the method trail as you extract, not afterward
PRISMA 2020 provides reporting templates for search, selection, and synthesis methods, including flow-diagram templates. Those templates are useful precisely because they force the method to be visible. Build the same visibility into your matrix.
Keep a running log with dated entries for:
- the search strategy and the sources used;
- the deduplication step;
- screening decisions at each stage;
- changes to the field definitions, with the reason for the change;
- the date of the last search rerun.
Cochrane recommends documenting the search process, deduplicating records, screening in stages, and rerunning searches before publication. A log that records these steps turns your matrix from a private working file into an auditable artifact. When a coauthor asks why a study was excluded, the answer is a dated entry, not a recollection.
Where does literature search fit into the repair?
If your diagnostic shows that the matrix is missing eligible studies, or that the search was never documented, the repair includes a search step. Plan it from the eligibility criteria rather than from keywords you happen to remember.
Cochrane's guidance favors combining free-text terms with subject headings and using multiple sources. That combination is what makes a search reproducible by someone else. When you need to run or extend that search, WisPaper search provides a scholarly search interface for finding academic papers, and WisPaper Scholar Search offers a public entry point for the same task. Record the strategy you used in your method log so the search can be repeated.
Turn search results into an organized, question-ready library
Search results are not yet a matrix. The intermediate step is a stable set of papers you can query and re-query as the fields evolve.
Save or upload the papers you are screening into My Library so the candidate set is organized in one place. Once papers are in the library, Library QA lets you ask questions based on papers uploaded to My Library or added from search results. That is useful during extraction: instead of re-reading a full report to confirm one field, you can ask a targeted question and then verify the answer against the source location you record in the matrix.
Keep the verification step explicit. The matrix cell should still point to where in the report the value came from, so a coauthor or reviewer can check it.
Handle citation integrity before the matrix feeds a manuscript
A comparable matrix is only as good as the references behind it. If a row's citation is wrong, the field values attached to it are attached to the wrong study.
Before the matrix feeds a bibliography, screen the BibTeX. TrueCite checks BibTeX against real academic databases to flag hallucinated references. Run it on the reference list that the matrix will generate, and resolve flagged entries before the manuscript circulates. Record the screening date in your method log alongside the search rerun date.
Coordinate the rebuild across coauthors and reviewers
A rebuild that lives in one person's files will drift. Put the project where collaborators can see the current state.
Research Projects is the WisPaper area for creating and running a project. Use it to hold the working context for the rebuild so the field definitions, the candidate set, and the extraction work sit together rather than in scattered folders. For longer tasks that fit a literature review or reproduction use case, Scholar Agent is the corresponding WisPaper area.
Agree on a small set of coordination rules:
- One owner for the field definitions; changes are logged with a date and reason.
- Two coders on a sample of rows, compared field by field, before the full extraction.
- A single place where the current matrix lives, with older versions archived rather than overwritten.
- A named reviewer for the citation screen and the search rerun.
These rules are what make the matrix survive a coauthor change or a lab handoff.
What does the corrected deliverable look like?
The repaired matrix is not a longer document. It is a structured one. A reader who has never seen the project should be able to open it and answer four questions without opening a single PDF:
- Which studies are included, and which reports belong to each study?
- For each included study, what are the values for every comparable field, and where in the source did each value come from?
- Why was each excluded record excluded?
- When was the search last run, and what changed since the previous run?
Alongside the matrix, the deliverable includes the method log, the eligibility criteria, the field definitions with their value sets, and the citation screening record. Together these are the evidence trail that a manuscript methods section, a peer-review response, or a lab handoff can draw on directly.
How do you know the matrix is ready?
Run a readiness check before the matrix leaves your hands. Each item below is a yes-or-no test, not a judgment call.
- Field completeness. Every included row has a value or an explicit "not reported" or "not checked" for every field.
- Location coverage. Every extracted value has a source location.
- Study-report linkage. Multiple reports of one study are linked to a single record.
- Exclusion reasons. Every excluded record has a recorded reason tied to a criterion.
- Method log currency. The log records the search strategy, deduplication, screening stages, field changes, and the date of the last rerun.
- Citation screen. The BibTeX has been screened and flagged entries resolved.
- Independent check. A second coder has compared a sample of rows field by field and the discrepancies are resolved or noted.
If any item fails, the matrix is not ready. The failure tells you which step to return to.
Transferable lessons for the next project
Three practices carry over to any evidence matrix, whether the next project is a manuscript revision, a new review, or a lab handoff.
Define fields before extracting. The cost of deciding comparable fields up front is small compared with rebuilding rows later. Derive them from eligibility criteria and from the synthesis you plan to run.
Treat every cell as a claim with a location. A value without a pointer to its source is a summary. The pointer is what makes the matrix auditable and what lets a coauthor check your work without re-reading the paper.
Log the method as you go. Search strategy, deduplication, screening stages, field changes, and rerun dates are cheap to record at the time and expensive to reconstruct. NIH guidance treats transparent methods, validation, and reproducibility as core parts of rigorous research practice; a method log is the concrete form that takes in a working matrix.
When you start the next project, the first artifact is the field definition sheet, not the first summary. The matrix that follows will be comparable from the first row.




