LHTB: Stress-Testing AI Agents on the Long-Horizon Frontier

Executive Summary

TL;DR: Researchers have released Long-Horizon-Terminal-Bench (LHTB), a benchmark designed to push AI agents beyond quick fixes into the realm of professional, multi-hour workflows. By replacing simple Pass/Fail metrics with dense, subtask-based rewards, LHTB reveals that even the most advanced models (like GPT-5.5) fail to fully complete more than 15.2% of complex tasks. The study identifies that the true bottleneck for current agents isn't just "intelligence" in the local sense, but the ability to sustain progress, manage time, and self-verify over hundreds of steps.

LHTB sits at the intersection of SOTA benchmarking and practical agentic engineering, serving as a "marathon" for models that have previously only run "sprints."

The "Success Mirage" in Agent Evaluation

Current benchmarks like SWE-Bench or the original Terminal-Bench often judge an agent by its final output. If an agent performs 99% of a task perfectly but trips at the finish line, it receives a score of 0. This "sparse reward" problem makes it impossible to distinguish a model that is "almost there" from one that is completely lost.

More importantly, real-world expert tasks—like reproducing a Nature paper's experiment or auditing a climate dataset—take hours, not minutes. The paper argues that we are currently experiencing a "long-horizon task mirage" where models look capable on short snippets but fall apart during sustained execution.

Methodology: Engineering Dense Feedback

The core innovation of LHTB is its Subtask-based Grading. Instead of one big check at the end, each of the 46 tasks is broken down into semantic milestones.

The reward $R$ is calculated as a weighted average of subtask scores:

• Binary Subtasks: Hard checks (e.g., "Did the script run?").
• Continuous Subtasks: Accuracy relative to a gold standard (e.g., "Is the reproduced figure value within 5% of the original?").
• Aggregated Subtasks: Performance over multiple episodes (e.g., "Win rate in a game").

This allows the benchmark to rank models based on Mean Reward, exposing a much richer hierarchy of capability than a binary leaderboard could.

Results: The Brutal Reality of Scale

The experimental results are a wake-up call for the LLM community. Even with a 90-minute time budget and millions of tokens at their disposal, agents struggle.

Key Findings:

1. GPT-5.5 Dominance: While it leads the pack, its 15.2% pass rate shows enormous "headroom" for improvement.
2. The "False Finish": Many agents suffer from a lack of self-verification. They complete 80% of a task, convince themselves they are done, and exit early while failing the hidden verifier.
3. Cost vs. Capability: Average costs per task reach up to $28. Interestingly, paying more for tokens does not always guarantee success; efficiency in planning matters more.

Deep Insight: Timeouts as the Primary Failure Mode

One of the most revealing parts of the study is the analysis of why models fail. In 79% of unresolved cases, the model timed out while still working.

This suggests that agents are not necessarily "stupid" in their individual actions; rather, they are inefficient marathons runners. They get stuck in loops, repeat redundant checks, or fail to manage their "mental budget" over long timeframes.

Conclusion and Future Outlook

LHTB proves that "Long-Horizon" is the next major frontier for AI agents. Passing LHTB won't just require a model that can code; it will require a model that can:

• Self-Correct: Recognize when an approach is wasting time.
• Verify: Conduct rigorous internal testing before declaring success.
• Conserve State: Maintain a clear "mental map" across thousands of tokens of CLI history.

As we move toward autonomous AI scientists and engineers, benchmarks like LHTB will be the primary filter for separating superficial chat-bots from truly reliable autonomous agents.