[USTC 2026] Utility-Guided Agent Orchestration: Balancing the Quality-Cost Trade-off in LLM Tool Use

TL;DR

As LLM agents become increasingly "agentic" by using tools and retrievers, they face a classic systems challenge: The Diminishing Returns of Reasoning. More steps often mean better answers, but at the cost of spiraling latency and token fees. This paper moves agent control from "implicit prompting" to an explicit decision problem, using a utility-guided policy to decide when to keep digging and when to stop.

Problem & Motivation: The "Over-Execution" Trap

In the current LLM landscape, we usually see two extremes:

1. Fixed Workflows: Predictable and cheap, but they break when the task gets hard.
2. Free-form Reasoning (e.g., ReAct): Highly flexible, but prone to "looping"—making redundant search calls or reasoning long after the answer is found.

The authors argue that Orchestration—the act of choosing the next move—is a control problem. Most agents use "vibes-based" control (prompting the model to stop when ready). This paper replaces that with a mathematical utility formula that forces the agent to justify every single token spent.

Methodology: The Utility Function

The core of the framework is a simple yet powerful utility equation for action selection:

$U(a|s_t)=extGain-{\lambda }_{1}extStepCost-{\lambda }_{2}extUncertainty-{\lambda }_{3}extRedundancy$

Instead of an LLM just "choosing" what to do, the system calculates scores for actions like retrieve, tool_call, and stop.

• StepCost: A penalty for trajectory growth.
• Redundancy: A crucial term that penalizes the agent for asking the same thing twice.
• Gain & Uncertainty: Self-estimated signals (from the LLM) about how much more information is needed.

Figure 1: The orchestration loop separates the environment, the state representation, and the decision policy.

Experiments & Results: Efficiency vs. Efficacy

The researchers tested this on HotpotQA. Key findings include:

• Workflow Inflexibility: Even "strong" fixed workflows (searching twice) failed to match adaptive agents, proving that static pipelines are fundamentally limited for complex QA.
• The ReAct Benchmark: While the ReAct method achieved higher raw F1 scores, the Utility Policy was much easier to "tune" for specific budgets (e.g., token-limited or latency-limited settings).
• Redundancy Control: By switching to semantic redundancy checks, the agent reduced tool calls from 1.56 down to 1.40 per task, saving significant context window space.

Table 1: The trade-offs between F1 quality, Token usage, and Wall-clock time.

Through Reasoning Depth Analysis, the authors visualize the point of diminishing returns. After 2-3 steps, the marginal gain in F1 score drops sharply while costs continue to rise linearly.

Figure 5: High self-estimated gain correlates strongly with the decision to continue, validating that these "internal" signals are behaviorally meaningful.

Critical Analysis & Conclusion

Takeaway

This paper is a wakeup call for developers building "Auto-GPT" style agents. You cannot simply rely on the model to "know when to stop." By implementing a lightweight utility layer, you can create agents that are defensible—meaning you can explain why the agent decided to call a certain API or stop its search.

Limitations

• Heuristic Nature: The "Gain" and "Uncertainty" are self-estimated by the LLM and not yet mathematically calibrated.
• Latency Overhead: While it saves tokens, the extra calculation of the utility score adds a tiny bit of latency (as seen in the wall-clock time results).

Future Outlook

The next logical step for this research is to learn the utility parameters via Reinforcement Learning (RL), allowing the agent to automatically find the "pareto-front" of the quality-cost curve based on real-world API pricing.