GRAFT: Amortizing 3D Human-Scene Optimization into Fast Transformers

TL;DR

Reconstructing a human standing on a floor or sitting on a couch from a single image sounds simple, but AI models usually fail at the "contact" part—humans either float or sink into the furniture. GRAFT (Geometric Refinement and Fitting Transformer) solves this by replacing slow, minutes-long optimization with a clever transformer that "probes" the 3D scene and corrects the human pose in milliseconds.

The "Floating Human" Problem

Current 3D Human Mesh Recovery (HMR) models are great at predicting a person's shape, but they are "scene-blind." When you place a predicted human mesh into a 3D scene reconstruction:

1. Floating: The feet don't touch the ground due to depth ambiguity.
2. Interpenetration: The person's back might overlap with a chair.

To fix this, researchers traditionally used optimization—calculating forces to "push" the human into the right place. But optimization takes 20+ seconds per image. Feed-forward models (like Human3R) are fast but lack the "spatial reasoning" to respect walls and floors.

Methodology: Probing the World with Tokens

The core insight of GRAFT is to treat the correction of the human pose as a learned gradient. Instead of calculating physics, the model looks at the gap between the person and the scene and predicts how to move the joints.

1. HSI Tokenization & Geometric Probes

GRAFT represents the human body using only 24 tokens (joints, hands, and full body). Each token performs a Geometric Probe:

• It finds the nearest point in the scene point cloud.
• It calculates the distance, direction, and "surface normal" (the angle of the floor/wall).
• This gives the model a direct sense of "touch."

2. The Recurrent Loop

Unlike one-shot models, GRAFT is recurrent. It makes a small update, "re-probes" the scene to see if the contact is better, and then updates again. This mimics the behavior of an optimizer but happens via a high-speed Transformer.

3. Fast Differentiable Scale Update

A major technical hurdle in 3D is "metric scale"—how big is the person relative to the room? GRAFT introduces a closed-form math solution to update the human's scale without re-running the heavy SMPL-X body model, keeping the training fast and differentiable.

Experiments: Performance vs. Speed

GRAFT was tested on challenging datasets like RICH and PROX. The results are striking:

• Efficiency: It is 50x faster than PhySIC (the previous gold standard for interaction quality).
• Accuracy: It improved Contact F1 scores by 113% compared to existing fast models like UniSH.
• Plug-and-Play: Even without looking at the image (Geometry-only mode), GRAFT can take a "bad" reconstruction from another AI and fix it, improving F1 scores by 44%.

Deep Insight: Why It Works

The success of GRAFT lies in its Inductive Bias. By forcing the Transformer to attend to "Geometric Probes" (metric 3D data) rather than just raw pixels, the model is forced to learn the laws of physics—that humans cannot occupy the same space as a couch. Because it uses sparse tokens, it generalizes to "in-the-wild" internet images much better than models that use dense, high-dimensional scene representations.

Conclusion & Future Work

GRAFT bridges the gap between the accuracy of classical geometry and the speed of modern deep learning. While it currently assumes scenes are static (it doesn't model a soft cushion deforming when you sit), it sets a new SOTA for physically-aware human reconstruction.

Key Takeaway: Recurrent refinement with 3D-anchored tokens is the "secret sauce" for making AI understand the physical constraints of our world.