FAPO: When Prompt Engineering Isn't Enough, Let the Agent Rewrite the Pipeline

TL;DR

Optimization in the LLM era has traditionally been synonymous with "prompt tuning." However, as pipelines grow into complex, multi-step workflows (RAG, agentic loops, etc.), the bottleneck is often the structure of the chain, not the text of the prompt. FAPO (Fully Automated Prompt Optimization) is a new framework that leverages Claude Code to act as an autonomous developer—inspecting failures at each step, and when prompts fail to move the needle, it refactors the pipeline code itself. It yields a massive +14.1 pp average improvement over existing SOTA optimizers like GEPA.

Background: The "Fixed Program" Wall

Most current optimization frameworks, such as DSPy, optimize prompts within a fixed program structure. While powerful, these systems are blind to "structural bottlenecks." For example, if a 3-hop retrieval chain simply cannot find the necessary Wikipedia articles for a fact-verification task, no amount of prompt "polishing" on the answer-generation node will solve the problem. The industry has lacked a tool that can look at the intermediate logs, realize "we need more retrieval hops," and then actually modify the code to add them.

Methodology: The "Prompt-First" Escalation Policy

FAPO treats an LLM pipeline as an inspectable LangGraph workflow. The secret sauce lies in its Attribution-Driven Optimization Loop.

1. The Multi-Step Attribution

Instead of just looking at the final "Pass/Fail" score, FAPO records the ChainState (inputs, outputs, and diagnostics) for every single node. This allows the step-attribution subagent to classify failures into specific buckets:

• Near-miss: Verbose answers that fail exact match.
• Abstention: The model refused to answer.
• Structural: Missing evidence from retrieval nodes.

2. The Scope Contract & Escalation

FAPO follows a disciplined hierarchy of interventions:

1. Level 1: Prompt Text (Always tried first).
2. Level 2: Chain Parameters (e.g., changing $k$ in retrieval).
3. Level 3: Chain Structure (Adding/removing nodes or changing logic).

Figure: The FAPO loop ensures that every code change is reviewed by a separate "Reviewer" agent to prevent data leakage and ensure scope compliance.

Experimental Showdown: FAPO vs. GEPA

The researchers tested FAPO across six diverse benchmarks and three task models (including GPT-5.4-mini and Gemma 3-12B).

The Structural Breakthrough

The most striking results occurred in HoVer (fact verification) and IFBench (instruction following). In these tasks, FAPO correctly identified that prompt-only search had plateaued.

• On HoVer: FAPO extended a 3-hop retrieval chain to 5 hops, boosting performance by up to +48 pp.
• On IFBench: It added deterministic post-processing nodes to enforce formatting, leading to a +38 pp gain.

Table: FAPO significantly outperforms GEPA (orange shading indicates statistically significant wins) by having the flexibility to change the pipeline structure.

Deep Insight: Model-Specific Optimization

One of FAPO's most interesting findings is that different models require vastly different "winning" prompts for the same task. For the CTIBench-RCM (security classification) task:

• GPT-5 thrived on a complex prompt with explicit NVD mapping rules (addressing its tendency to confuse specific CWE categories).
• Foundation-Sec-8B-Instruct, conversely, actually performed worse with complex rules. FAPO discovered that a 2-line minimalist prompt was optimal for this smaller model, reducing noise and improving instruction following.

Critical Analysis & Takeaways

FAPO represents a shift from "LLM as an Optimizer" to "LLM as a Research Engineer."

• Statistically Robust: By reporting multi-trial means and standard deviations, the authors highlight a key reality—structural optimization has higher variance (path dependence) because discovering a structural "pivot" is more transformative than incremental prompt edits.
• The "Claude Code" Advantage: The success of FAPO is heavily tied to the reasoning capabilities of its orchestrator. Using a PhD-level coding agent allows for high-fidelity code modifications that were previously too risky for automated systems.
• Future Impact: This work suggests that we should stop building "static" pipelines. Future AI systems should likely ship with an "optimization tenant" that allows the system to self-correct its own architecture as it encounters new failure modes in production.

Conclusion

FAPO proves that the most effective way to optimize an AI system is to give the optimizer the same tools a human developer has: access to the codebase, the ability to read the logs, and the authority to refactor the architecture. It effectively bridges the gap between prompt engineering and software engineering.