[arXiv 2025] SEARCH-P1: Beyond Binary Rewards—Stabilizing Agentic RAG with Path-Centric Learning

TL;DR

Training autonomous agents to "search and think" has traditionally been a struggle between sparse rewards and high failure rates. SEARCH-P1 changes the game by introducing Path-Centric Reward Shaping. Instead of just checking if the final answer is right (binary outcome), it rewards the model for following a logical reasoning path. This leads to a +7.7% accuracy boost on public benchmarks and a massive 20.6% jump in industrial multi-hop QA tasks.

The Problem: The "All-or-Nothing" Reward Trap

In Agentic RAG, an LLM must decide when to retrieve information and how to synthesize it. Most current Reinforcement Learning (RL) methods use Outcome Rewards: if the final answer matches the ground truth, $R=1$; otherwise, $R=0$.

This approach has three fatal flaws:

1. Reward Sparsity: Intermediate reasoning steps (like a brilliant search query) get no credit.
2. Low Sample Efficiency: A trajectory that is 90% correct but misses one character in the answer contributes zero learning signal.
3. Slow Convergence: The model wanders aimlessly in the early stages of training because it rarely "hits" the correct answer by chance.

Methodology: The Path-Centric Breakthrough

The core philosophy of SEARCH-P1 is that the journey is as important as the destination. The framework restructures the reasoning trajectory to include an explicit Planner ($p$), making the model's intent observable.

1. Dual-Track Path Scoring

To give the model a dense signal, SEARCH-P1 evaluates the trajectory from two angles:

• Self-Consistency (Track A): Does the model actually follow the plan it set for itself? This measures execution reliability.
• Reference-Alignment (Track B): How well does the model's path cover the essential steps identified by a high-capability "Expert" model (Reference Planner).

The final path reward is the maximum of these two, ensuring that if a model finds a better way than the expert, it isn't penalized for straying.

Figure 1: Overview of SEARCH-P1 framework featuring Dual-Track Path Scoring and Soft Outcome Scoring.

2. Soft Outcome Scoring

Even when the final answer is wrong, SEARCH-P1 extracts value. It uses an LLM evaluator to provide "partial credit" based on how close the answer was and how sound the reasoning process remained. This converts "failed" samples into meaningful training data.

Experimental Results: Industrial Strength Performance

The authors tested SEARCH-P1 on seven public benchmarks and a proprietary internal dataset (AD-QA) focused on complex advertising logic.

• SOTA Gains: Outperformed Search-R1 and HiPRAG consistently across 3B and 7B scales.
• Efficiency: Reached the baseline's final accuracy 2.5x faster (60 steps vs 150+ steps).
• Robustness: The method works whether you use GRPO or PPO, and is independent of the specific base model (Qwen vs Llama).

Figure 2: Performance across various QA benchmarks. SEARCH-P1 (orange) consistently leads.

Ablation Insight: Why it works

The ablation study confirmed that Reference-Alignment is the most critical component. Without it, the model lacks the "expert guidance" needed to structure complex multi-hop queries. Furthermore, the Soft Format Reward ensures that the model doesn't get stuck in the early stages simply because it forgot to close a tag like </reasoning>.

Real-World Case Study

In a multi-hop query about a specific album ("Bang Bang Rock & Roll"), older models like Search-R1 mistakenly searched for the genre "rock & roll," leading to irrelevant noise. SEARCH-P1’s path-centric training forced the model to plan first, correctly identifying the album as a single entity and formulating a precise query.

Figure 3: Contrast in reasoning quality between Search-R1 and SEARCH-P1.

Conclusion & Future Outlook

SEARCH-P1 demonstrates that we can't treat agent training like a simple game of "right or wrong." By shaping the reward landscape to favor structural quality and process and consistency, we can build agents that are not only more accurate but also more efficient searchers.

Takeaway: For researchers and engineers building Agentic RAG, the message is clear: Invest in intermediate signal extraction. Dense, path-aware rewards are the key to moving beyond toy examples and into the complexities of industrial-grade knowledge synthesis.