[Tencent Robotics X] HY-Embodied-0.5: Bridging Digital Intelligence and Physical Action

TL;DR

Tencent's new HY-Embodied-0.5 suite tackles the "reality gap" in robotics by introducing a family of models (2B and 32B) specifically optimized for spatial reasoning and action planning. By utilizing a Mixture-of-Transformers (MoT) architecture and an iterative post-training pipeline with RL and distillation, they've created an edge-ready 2B model that punches far above its weight class, outperforming competitors twice its size.

Background: The Perception-Action Gap

While modern VLMs (like GPT-4o or Gemini) are incredibly "smart" in digital conversation, they often struggle when asked: "How many centimeters is that cup from the edge of the table?" or "What is the 3D trajectory needed to hang this mug?". This failure stems from a lack of fine-grained spatial perception and embodied grounding. Most current models treat images as flat data rather than 3D environments.

Methodology: Specialized Architecture for Special Tasks

The HY-Embodied team introduces three critical architectural innovations to solve this:

1. Mixture-of-Transformers (MoT)

Unlike standard "Dense" models where every token passes through the same parameters, MoT introduces modality-specific parameters.

• Visual Tokens use a dedicated branch with bidirectional attention to capture global context.
• Text Tokens use standard causal attention. This effectively doubles the visual modeling capacity without slowing down inference on edge devices.

2. Visual Latent Tokens

The model appends "latent tokens" to the end of visual sequences. These tokens are trained to act as a bridge—compressing high-dimensional visual information into a format that the Language Model part of the brain can "think" about during its Chain-of-Thought (CoT) process.

3. Iterative Self-Evolving Training

To cultivate "Deep Thinking" in robots, the researchers used a three-step post-training process:

1. Reinforcement Learning (RL): Using a GRPO-based objective to explore better action sequences.
2. Rejection Sampling (RFT): Filtering the best reasoning traces to "consolidate" successful behaviors.
3. On-Policy Distillation (OPD): Forcing the 2B "student" model to match the token-level distribution of the 32B "teacher" on its own mistakes.

Experiments: SOTA Achievement

The model was put through the wringer across 22 public benchmarks.

• Perception: In depth estimation and 3D spatial reasoning (3DSRBench), the 2B model consistently beat specialist models like RoboBrain.
• Efficiency: The MoT structure showed faster convergence during training while maintaining near-identical inference speed compared to standard dense layers.
• Robot Control: When deployed on physical hardware, the VLA version of HY-Embodied-0.5 achieved a 75% success rate on the "Mug Hanging" task, nearly doubling the rate of the π0 baseline (45%).

Deep Insight: Thinking Before Acting

One of the most impressive parts of the paper is the visualization of the Chain-of-Thought (CoT). When asked a spatial question, the model doesn't just output a coordinate. It "talks" to itself:

"Wait, the starting position is the chair... to get to the door, the robot needs to turn back first."

This "Deep Thinking" allows the model to catch its own errors before committing to a physical action, a capability previously seen only in massive frontier LLMs.

Conclusion and Future Outlook

HY-Embodied-0.5 proves that architecture matters. By tailoring the transformer block to handle vision and language differently, and by focusing the training data on spatial and geometric metrics, Tencent has provided a blueprint for the next generation of "Robot Brains." The next frontier? Closing the gap between these high-level planners and the low-level millisecond-latency motor controllers.

Takeaways

• Decouple your modalities: MoT is superior to unified dense transformers for embodied tasks.
• Small models can think: Through on-policy distillation, 2B models can inherit "reasoning styles" usually reserved for 30B+ models.
• Spatial data is king: Pre-training on depth, 3D grounding, and trajectories is non-negotiable for embodied agents.