OPENFORGE RL: Mastering AI Agents via Harness-Native Training

TL;DR

SOTA AI agents are no longer just "raw" models; they are systems wrapped in complex inference harnesses. However, training these agents has traditionally ignored this layer, causing a performance gap. OPENFORGE RL bridges this by allowing models to be trained directly within deployment scaffolds like OpenClaw and Codex. By scaling rollouts in Kubernetes containers, it produces open-source agents that outperform models many times their size.

The Hidden Layer: Why "Bare" Training Fails

If you look at the top of the leaderboards for coding or web navigation, you'll see Claude Code or OpenAI's Codex. These aren't just LLMs; they are agents equipped with harnesses—complex software layers that manage state, sub-agents, and persistent tool contexts.

The problem? Current RL frameworks like veRL or DeepSeek-R1's pipeline expect a simple prompt-response loop. They can't handle a harness that spawns three different sub-processes and manages a 10-turn conversation internally. Researchers usually simplify the harness for training, but this leads to a train-deploy mismatch: the model learns to solve tasks in a way that doesn't work when it's finally plugged into the real Claude Code or OpenClaw interface.

Methodology: The Hook and the Cloud

OPENFORGE RL introduces a "plug-and-play" architecture to solve this, focusing on two key innovations:

1. The Lightweight Proxy: Instead of the trainer calling the model, the harness calls the model. The OPENFORGE proxy intercepts these OpenAI-style API calls, handles the generation via vLLM, and reconstructs the multi-turn interaction into a standard RL trajectory.
2. Kubernetes Orchestration: GUI and Coding tasks are resource-heavy. You can't run 50 Chrome browsers on the same GPU node where you're training a 30B model. OPENFORGE offloads rollouts to remote Kubernetes pods (using Microsoft Azure), allowing for massive horizontal scaling of "on-policy" experiences.

Experiments: Punching Above Their Weight

The authors tested the framework across two major domains:

• Claw Agents (Text/Tool Use): OpenForge-Claw (a 30B MoE) achieved 31.7 (pass3) on ClawEval, significantly beating the base Qwen3 model (14.3). It even generalizes to unseen harnesses, proving that training on multiple scaffolds (ZeroClaw + Codex + OpenClaw) makes the model more robust.
• GUI Agents (Vision): OpenForge-GUI (8B) reached 72.3 on WebVoyager, matching models like Qwen3-VL-235B. This is remarkable because it used only a few thousand tasks, whereas competitors often use hundreds of thousands.

Deep Insight: What does RL actually teach an Agent?

One of the most profound sections of the paper explores the behavioral shift after RL. By training "in-harness," the authors found that RL doesn't just help the model "find the answer." Instead, it improves agentic reliability:

• Self-Verification: The model learns to "read back" its work (e.g., checking a file after writing to it).
• Tool Specialization: RL discourages the model from using generic tools (like a bash shell) when specialized tools (like an Email API) are more efficient.
• Error Recovery: While still the weakest link, RL-trained models are significantly better at trying a second approach if the first tool call fails.

Critical Analysis & Conclusion

OPENFORGE RL is a major step toward democratizing high-end agent training. By treating the harness as a "first-class citizen" in the training loop, it aligns model behavior with real-world deployment.

Limitations: The system still struggles with Error Recovery—the ability to pivot gracefully when an environment throws an unexpected error. This suggests that while RL optimizes "success," we might need more specialized "failure-recovery" datasets to make agents truly autonomous.

Future Work: The framework is now open-source, and its ability to pair "any harness with any environment" opens the door for researchers to explore more complex multi-agent architectures (sub-agents calling sub-agents) in a scalable, verifiable way.