LiveResearchBench: Moving AI from Searchers to Real Researchers
LiveResearchBench: A Live Benchmark for User-Centric Deep Research in the Wild
LiveResearchBench is a new benchmark featuring 100 expert-curated, dynamic, and user-centric tasks designed to evaluate the capability of AI agents to perform "Deep Research." Alongside this, the authors introduce DeepEval, a multi-dimensional evaluation suite that sets a new standard for assessing long-form, citation-grounded reports using an Agent-ensemble-as-a-judge approach.
TL;DR
Addressing the "benchmarking crisis" in agentic systems, LiveResearchBench introduces 100 high-difficulty, real-time research tasks paired with DeepEval, a six-dimensional evaluation framework. The study reveals a sobering reality: while current AI agents (like o3, Gemini, and Specialized MAS) can "find" information, they struggle to "research" it—often failing at citation accuracy and failing to synthesize deep, logical insights.
The Motivation: Why Current Benchmarks Fail
Most existing benchmarks test "fact retrieval" or "closed-ended" questions. Even those targeting "Deep Research" often suffer from:
- Static Content: Proximity to training data makes them easy to "memorize."
- Ambiguity: Lack of target audience or report format leads to inconsistent model outputs.
- Evaluation Gaps: Using a single LLM as a judge leads to high variance and "conflated scores" (where a model likes a report's style and ignores its factual errors).
LiveResearchBench addresses this by requiring real-time search (time-varying) and providing human-verified checklists for every task.
Methodology: DeepEval’s Multi-Faceted Calibration
The core innovation in evaluation is DeepEval, which breaks down report quality into six axes. Critically, it refuses to use a "one-size-fits-all" score.
| Metric | Protocol | Reasoning |
|---|---|---|
| Coverage | Checklist-based | Uses human-curated "unit tests" for each prompt requirement. |
| Consistency | Pointwise (Additive) | Penalties are subtracted for every contradiction found. |
| Analysis Depth | Pairwise Comparison | Direct head-to-head comparison to identify "insight density." |
| Citation Accuracy | Rubric Tree | Validates URLs for accessibility, relevance, and support. |
Insights: The SOTA Gap
The authors evaluated 17 systems, including OpenAI o3, Gemini Deep Research, and various Multi-Agent Systems (MAS) like OpenManus.
1. The Multi-Agent Advantage vs. The Single-Agent Polish
Multi-agent systems (like Open Deep Research) generally outperformed single-agent systems in Presentation and Citation Association. This is because MAS often feature specialized "citation agents" or "checkers" that clean up the output. However, Single-Agent Web models (like Gemini 2.5 Pro) actually maintained better Factual Consistency, likely because they operate on a single, continuous memory stream without "logical drift" during agent hand-offs.
2. The "Deep Searcher" Trap
Perhaps the most critical finding is shown in the Analysis Depth win rates. Even SOTA models that generate extremely long reports (e.g., o3 Deep Research) often behave as efficient "collectors." They organize information well (high coverage) but fail to create "causal chains" or "multi-layer insights"—the hallmarks of true research.
3. Citation Hallucination is Still Rampant
Using the Rubric Tree (E1: Inaccessible, E2: Irrelevant, E3: Unsupported), the researchers found that even GPT-5 and Grok-4 produce non-trivial errors. Most common are E3 errors—where a URL is valid and relevant, but the actual content does not support the specific claim the model made.
Critical Analysis & Future Outlook
The paper identifies a "trade-off" bottleneck: Coherence vs. Verifiability vs. Breadth. As models include more web sources, they often hit context limits, leading to:
- Information Loss: Dropping key evidence to fit the window.
- Logical Drift: Inconsistencies between the beginning and end of long reports.
Takeaway: Future research shouldn't just focus on "more search." The next leap in AI agent capabilities will require hierarchical information compression (summarizing without losing evidence) and long-horizon memory that updates as the research progresses.
LiveResearchBench provides the "stress test" needed to push models past the stage of simple chatbots toward becoming truly independent, reliable problem-solvers.