I Tried Three Paper Search Tools on the Same Question. What Did Each Miss?

I gave OpenAlex, Crossref, and Europe PMC the same query: digital cognitive behavioral therapy for insomnia in university students. Then I manually checked the first ten results from each tool.

This was a small search audit, not a benchmark. Still, it exposed something useful: the tools did not merely rank the same papers in a different order. They built three different candidate sets.

OpenAlex put the most directly relevant university-student paper first and found a college email intervention in its top ten. It also widened the set to depression, heavy drinking, diabetes, and general adherence studies.

Crossref found the exact target paper first, but several of its next results were book chapters or front matter with titles such as “Copyright,” “Index,” and “Acknowledgments.” Crossref is excellent for confirming publication metadata; a broad relevance-ranked query can still need heavy screening.

Europe PMC returned recent health literature. It found the target commentary and related student sleep studies, but it also surfaced nomophobia, social-media use, PTSD, and bibliometric work. Its medical scope was useful, yet the wording of the query was not enough to keep every result on the intervention question.

The practical result was simple: no top-ten list was a finished reading list.

The misses fell into three groups:

A search answer can look polished while hiding all three. That is why result count is a poor measure of search quality. I care more about whether the set contains known relevant papers, useful new papers, and enough variety to reveal missing terms.

Use the same question, date, and screening rules. Export or record the first 10–20 results, then label each one:

Also keep a small gold set of papers that a competent search should find. A tool that returns many plausible titles but misses two obvious seed papers needs a revised query.

The three raw result sets from this test were saved before I judged them. That matters because search results change, and memory tends to make the tool we liked feel more complete than it was.

Use WisPaper Search to run the research question in natural language, then screen the paper cards with the same labels. Save the relevant set to My Library and use Library QA to ask which populations, interventions, and outcomes are actually represented.

The useful handoff is not “one tool found the truth.” It is: one search creates candidates, screening removes scope errors, and a second route looks for what the first route missed.