StateVLM: Bridging the Gap Between Visual Perception and Robotic Affordance Reasoning
StateVLM: A State-Aware Vision-Language Model for Robotic Affordance Reasoning
The paper introduces StateVLM, a state-aware Vision-Language Model specifically designed for robotic affordance reasoning and fine-grained object-state localization. By integrating an Auxiliary Regression Loss (ARL) during training, the model achieves state-of-the-art performance on referring expression comprehension tasks and a newly proposed benchmark, OSAR.
TL;DR
Researchers from the University of Hamburg and KAUST have unveiled StateVLM, a vision-language model that masters the "where" and "state" of objects (e.g., "is the plate dirty?"). By introducing a novel Auxiliary Regression Loss (ARL), the model overcomes the inherent numerical weakness of LLMs. It not only localizes objects but also identifies specific graspable regions based on their current physical state—a critical requirement for reliable robotic manipulation.
Background: Why Robots Struggle to "See" Numerically
Most modern Vision-Language Models (VLMs), such as LLaVA or Shikra, treat object localization as a translation task. They convert bounding box coordinates (x, y) into discrete text tokens (e.g., [12, 453, 98, 201]). While this fits the architecture of Large Language Models (LLMs), it is fundamentally "leaky." Modeling continuous space with discrete symbols is inefficient.
Furthermore, in robotics, identifying an object isn't enough. A robot needs to know:
- Object State: Is this knife dirty? (If yes, don't touch the blade).
- Fine-grained Affordance: Where is the handle vs. the content?
Existing benchmarks often ignore these dynamic properties, leading to the creation of the Object State Affordance Reasoning (OSAR) benchmark.
Methodology: The "Secret Sauce" of ARL
The core contribution of StateVLM is its training strategy. While most models stick to either Pix2Seq (text tokens) or Pix2Emb (embeddings), StateVLM proposes a less intrusive "Dual-Track" supervision:
- Training Phase: The model uses a standard LLM backbone (MiniCPM-V) but attaches a lightweight Box Decoder. This decoder takes the hidden states and produces continuous values for an Auxiliary Regression Loss (ARL). This loss combines L1 loss (for coordinate accuracy) and GIoU loss (for shape consistency).
- Inference Phase: The decoder is discarded. The model outputs standard text, but because it was supervised by the regression loss during training, its internal "understanding" of coordinates is far more precise.
Figure 1: The dual-phase workflow of StateVLM. (a) Training with ARL; (b) Inference via standard sequence prediction.
The OSAR Benchmark: Testing the Limits
To evaluate state-aware reasoning, the authors generated 1,172 scenes using Stable Diffusion and Blender. Unlike RefCOCO, which focuses on labeling "the red cat," OSAR asks more complex questions:
- Instruction: "Hand me the drink."
- Reasoning: The model must find the glass (object) and the specific region part of the glass not containing the liquid (grasping area).
Figure 2: Diversity of objects and states in the OSAR benchmark, ranging from hygiene (clean/dirty) to contents (empty/full).
Experiments & Results: Consistency is Key
The results confirm that ARL provides a significant boost. On the standard RefCOCO datasets, StateVLM (LCLM+ARL) outperformed the baseline by 1.6%. However, the real victory was in Consistency.
In complex affordance tasks, standard VLMs often "break," outputting malformed strings or failing to understand the task. StateVLM with ARL achieved a 0% exception rate, whereas the version without ARL failed to produce a valid bounding box format over 33% of the time in affordance reasoning.
Figure 3: Comprehensive performance showing how ARL (yellow) consistently elevates accuracy across detection and reasoning.
Critical Insight: The "Why" behind the "How"
Why does adding a loss that isn't even used at runtime work so well? It acts as a structural regularizer. By forcing the LLM's hidden states to be decodable into continuous coordinates, the model learns a geometrically grounded representation of features. It moves away from mere pattern matching of text tokens toward a genuine spatial understanding of the image.
Limitations and Future Work
While StateVLM is a leap forward for state-awareness, it still struggles with thin objects like cutlery. Future research needs to explore even higher resolution visual encoders and ambiguous instruction handling (e.g., "Give me the one I used earlier").
Conclusion
StateVLM proves that if we want robots to understand the world, we must move beyond the "text-only" mindset of LLMs. By supervising the latent space with regression losses, we can build models that are not just good at chatting, but also precise enough to assist in a messy, dynamic kitchen environment.