Anny-Fit: Solving the All-Age Ambiguity in 3D Human Mesh Recovery

TL;DR

Anny-Fit is a novel optimization framework that moves beyond "adult-only" 3D human reconstruction. By combining specific expert cues—metric depth, instance segmentation, and high-level semantic data from Vision-Language Models (VLMs)—it recovers accurate 3D meshes for everyone from infants to seniors. It transforms the challenging problem of depth-scale ambiguity into a solvable joint optimization, enabling even adult-trained models to reconstruct children correctly in "zero-shot" settings.

The "Small Person" Paradox: Why All-Age HMR is Hard

In monocular 3D vision, distance and size are inextricably linked. For years, HMR researchers took a shortcut: they assumed every subject was an adult. Under this assumption, a small figure in an image is simply a distant figure.

However, in a real-world scene with families, this logic collapses. Is that a 6-foot adult standing 10 meters away, or a 3-foot toddler standing 5 meters away? Without knowing the age and proportions of the subject, the depth is mathematically impossible to resolve perfectly. This is the depth-scale ambiguity.

Methodology: Expert-Guided Optimization

Anny-Fit addresses this by treating reconstructive HMR as an expert-guided optimization problem. Instead of relying on a single neural network to "guess" everything, it orchestrates several "expert" models:

1. The VLM Expert: Models like Qwen2.5-VL are prompted to estimate age and gender. These semantic labels are mapped to the Anny body model's shape space.
2. The Depth Expert: Off-the-shelf metric depth estimators provide ordinal cues to ensure that if Person A is behind Person B in the real world, their 3D meshes reflect that.
3. The Geometry Experts: 2D keypoint and dense point regressors ensure the mesh "sticks" to the 2D image evidence.

Multi-Person Joint Fitting

Unlike traditional methods that crop each person and process them in isolation, Anny-Fit performs Multi-Person Optimization. It fits all individuals simultaneously in the camera coordinate system. This prevents people from "floating" at inconsistent depths and ensures the entire scene makes physical sense.

Experimental Breakthroughs

The authors tested Anny-Fit on challenging datasets like Relative Human and CMU Panoptic Toddler. The results were striking:

• Zero-Shot Adaptation: Adult-only models (like CameraHMR) could suddenly reconstruct toddlers with high fidelity without any retraining.
• Metric Gains: The system reduced 3D joint errors by up to 29mm and improved age/gender classification F1 scores by over 80 points in some cases.
• Distillation Power: The high-quality outputs (pseudo-ground truth) from Anny-Fit were used to re-train faster feedforward models, extending the knowledge of "what a child looks like" into existing HMR architectures.

Critical Insight: Semantic to Geometric Bridging

The most profound takeaway from the Anny-Fit paper is the effective use of VLMs as geometric priors. While VLMs can't output 3D coordinates directly (yet), they are excellent at categorizing human attributes. By mapping the word "toddler" to a specific zone in a continuous 3D shape space, the authors have bridged the gap between language understanding and spatial geometry.

Conclusion & Future Work

Anny-Fit successfully navigates the ethical and practical hurdles of all-age HMR by using "experts" instead of massive 3D child-scan datasets. While it still struggles with extreme occlusions and doesn't yet explicitly model physical "interpenetration" (people clipping through each other), it provides a robust blueprint for human-centric AI that truly understands the diversity of the human body across a lifespan.