[TOPReward] Beyond Text: Using Token Probabilities as Zero-Shot Rewards for Robotics

TL;DR

Training reward models for robotics usually requires massive datasets or proprietary APIs. TOPReward changes the game by proving that open-source Vision-Language Models (VLMs) like Qwen3-VL already "know" when a task is progressing—you just have to look at their internal token probabilities instead of their text output. By extracting the probability of the token "True" in response to a completion query, the authors achieve SOTA zero-shot progress estimation across 130+ real-world tasks.

The Motivation: The "Representation Bottleneck"

Reinforcement Learning (RL) in the real world is notoriously difficult because of "sparse rewards"—the robot only knows it succeeded at the very end. While we want "dense rewards" (a score that increases as the robot gets closer to the goal), current methods are stuck:

1. Fine-tuned Models: Require thousands of hours of robot data and don't generalize to new environments.
2. VQA-based Prompting (GVL): Asks a VLM "What is the progress percentage?". Unfortunately, open-source LLMs are terrible at math and numerical calibration. They might understand the video but can't express the progress as a stable number.

TOPReward identifies that the failure of open-source models isn't a lack of visual understanding, but a generation bottleneck.

Methodology: Probing the Latent Belief

Rather than forcing a VLM to "think" of a number, TOPReward uses a simple binary prompt:

"The above video shows a robot... that completes the following task: {Instruction}. The answer is: True"

The core innovation is calculating the log-probability of the token "True": $r_t=\log p_{h}eta(extTrue\mid extVideo,extPrompt)$

As the robot moves through the trajectory (prefixes $a{u}_{1:t}$), the model's confidence in the "True" token naturally increases. This signal is then normalized to a [0, 1] range to represent task progress.

Figure: TOPReward tracks completion evidence in real-time, showing monotonic increases as subtasks (like grasping and folding) are completed.

ManiRewardBench: A New Gold Standard

To test this, the authors introduced ManiRewardBench, featuring:

• 130+ tasks (Stacking, sorting, pouring, folding).
• 4 robot platforms (Franka, YAM, SO-100/101).
• Stage-aware annotations: Precise timestamps for subtasks to measure how granular the reward signal truly is.

Experimental Results: Dominating Open-Source

The results show a massive performance gap between the old prompting style (GVL) and TOPReward. On the Qwen3-VL-8B model:

• GVL VOC: ~0.194 (Nearly random ordering)
• TOPReward VOC: 0.947 (Near-perfect correlation with time)

Notably, TOPReward also solves the "Success Detection" problem. Previous metrics like VOC only care if the sequence is ordered; a robot that fails halfway but stays still would still get a high VOC. TOPReward, measuring absolute probability, correctly assigns low scores to failed attempts (improving ROC-AUC by +0.135).

Figure: TOPReward (orange) vs GVL (blue) across OXE and ManiRewardBench datasets.

Real-World Impact: Policy Improvement

The authors didn't just stop at evaluation. They used these rewards to weight expert data for Advantage-Weighted Behavior Cloning. In real-world tests on the SO-100 robot, this "Reward-Aligned" training allowed the robot to master tasks where standard imitation learning failed.

Figure: TOP-AWR (Ours) successfully placing a doll in a box, whereas the base policy fails.

Critical Analysis & Conclusion

The Takeaway: TOPReward proves that the "World Knowledge" inside VLMs is more fine-grained than the text they generate. By looking at logits, we find a "free" reward model that generalizes across robots and tasks.

Limitations:

1. Normalization: The signal is currently normalized per-episode, meaning you can't easily compare absolute progress between two different types of tasks without a baseline.
2. Backbone Dependency: The reward quality is strictly capped by the VLM's visual acuity—if the model can't see a small screw, it can't reward the robot for turning it.

Future Work: This method opens the door for online RL on real robots using nothing but a pretrained, off-the-shelf VLM as the judge.