How can fine-grained 3D understanding be improved through target-conditioned visual foveation?

Why 3D LLMs miss fine-grained detail

Existing 3D large language models such as Video-3D LLM and 3DRS can locate objects in indoor scenes, but the paper reports that they often overlook fine-grained attributes and less visually salient objects and parts, even when the relevant evidence is present in the scene video. The paper's example is a model that locates an overhead cabinet yet struggles to identify its door material or handle orientation.

The paper attributes this gap to two factors. First, existing 3D datasets emphasize object categories, referring expressions, and short descriptions, so they provide limited supervision for fine-grained attributes. Second, the relevant details may not be visually salient, and their visibility varies across viewpoints, so even when captured in the video they can be hard for a model to distinguish from surrounding content and use for the task.

LensUnd grounds, focuses, then understands

Lens3D's external understanding pipeline, LensUnd, follows a Ground-Focus-Understand scheme. A 3D grounder G predicts a bounding box b for the object or region relevant to a language query q. The Focus stage then selects complementary RGB views in which the target region is explicitly marked, and an external 2D vision-language model U processes those views under a task-specific instruction p to produce a response r.

The same evidence-collection procedure supports three tasks by varying the instruction and output format: fine-grained object captioning, small-object grounding, and fine-grained object question answering. For small-object grounding, the initially grounded region can contain the smaller object of interest, so LensUnd combines per-view 2D predictions into a 3D box using enclosed depth and camera parameters. LensUnd requires only the predicted box from G and does not modify the underlying grounder or VLM.

View selection maximizes new target surface

The Focus stage does not simply pick the highest-quality frames. It first builds a candidate set C of geometrically valid, depth-visible frames and assigns each a quality score st based on projected target completeness, image coverage, centrality, and occlusion estimated through depth consistency. It then unprojects depth pixels, retains 3D points inside the target box b, and discretizes them into an occupied voxel set Vt on a shared grid with roughly nv = 12 cells along the shortest box axis. Vt therefore represents observed surfaces within the target region rather than its projected image area.

Selection starts with the highest-quality frame and then greedily maximizes additional surface coverage, measured as the voxels in Vt not already covered by selected frames. Selected frames must be separated by at least delta frames, and when no feasible candidate adds new voxels the remaining slots are filled by quality under the same temporal constraint. The paper calls this strategy bbox-voxel and uses K = 4 views and delta = 12 by default. Each selected RGB image is marked by projecting the twelve edges of b as a wireframe, giving the 2D VLM an explicit instance-level spatial condition so it can identify the object to describe when multiple same-class objects are visible.

LensDistill moves knowledge into native inputs

LensUnd relies on the full RGB-D sequence, target-conditioned view selection, and an additional 2D VLM call, which introduces inference cost if deployed directly. LensDistill avoids that cost through asymmetric privileged-information distillation. During training, the teacher accesses privileged target-level evidence Eb, the wireframe-marked views, and generates a fine-grained understanding y-star under instruction p. The student keeps the base 3D LLM's native scene input X, the target center c(b) encoded by the <coord> token, and instruction p.

The student is trained with an autoregressive objective on the teacher's detailed captions. Because the selected, wireframe-marked views are used only to generate teacher supervision, LensDistill transfers knowledge acquired from privileged target-level observations rather than matching teacher and student responses under identical inputs. After distillation, the student generates detailed descriptions directly from X, c(b), and p without additional view processing or 2D VLM calls. The paper validates this transfer paradigm specifically on fine-grained object captioning, using structured, detailed captions as supervision.

LensBench provides instance-level references

Evaluating fine-grained captioning requires references that capture instance-level details, which the paper says existing captioning benchmarks cover sparsely. The authors therefore construct LensBench, a held-out evaluation set of 2,068 objects with three independently generated VLM descriptions per object. A questionnaire-based quality audit supports their use as silver-standard references.

The silver-standard label matters for interpretation: these references are model-generated descriptions validated by a quality audit, not human-authored gold captions. Results measured against LensBench therefore reflect agreement with that reference construction, and the paper's own framing treats them as silver-standard rather than definitive ground truth.

Captioning gains without losing grounding

Experiments with two representative 3D LLMs, Video-3D LLM and 3DRS, show that LensDistill substantially improves fine-grained object captioning while preserving existing grounding and scene-level QA performance. The paper reports that captioning reaches performance comparable to a similarly sized 2D VLM. In a blind user study on 100 objects using a four-point scale, LensDistill raised the mean rating from 1.92 to 3.86 for 3DRS and from 1.88 to 3.71 for Video-3D LLM.

The retained-capability result is reported under the detailed-captioning instantiation of LensDistill: the method enhances the fine-grained description capability that is limited in the base models while preserving, and in some cases improving, their original grounding and question-answering performance, with gains most apparent in grounding. These are separate measures from captioning quality, so the captioning improvement should not be read as evidence about grounding, or vice versa.

What drives the gains and what does not

Ablations isolate several contributors. Wireframe marking increases Summary F1 from 0.263 to 0.370 with the selected views. With ground-truth boxes, four views, projected wireframes, and the same VLM, bbox-voxel frame selection reaches a Summary F1 of 0.445, outperforming ExCap3D's visibility ranking at 0.436 and Video-3D LLM's greedy selection used in object-level at 0.353. Scaling the VLM from 2B to 8B further improves Summary F1 from 0.445 to 0.485 and Attr. Mean from 0.397 to 0.528, with corresponding gains for parts, materials, and states of 22.2, 9.4, and 7.6 percentage points.

Two negative or robustness findings bound the method. Increasing temporal uniform sampling from 4 to 32 views yields only marginal gains, possibly because the additional views provide little target-relevant evidence, which supports selecting views by target coverage rather than by count. Using predicted boxes instead of ground-truth boxes retains at least 97.5% of the performance across the main metrics, so the pipeline is reported as robust to predicted target boxes.

Where the transfer claim holds

The paper's central claim is feasibility: externally acquired fine-grained knowledge can be transferred into native 3D LLMs. That claim is established for fine-grained object captioning, on indoor scenes, with two base models, Video-3D LLM and 3DRS, and evaluated against LensBench's silver-standard references. LensUnd's other two tasks, small-object grounding and fine-grained object QA, are illustrated qualitatively rather than quantitatively evaluated in the reported results.

The practical priority that follows is the trade the paper makes explicit. LensUnd buys fine-grained detail at the cost of the full RGB-D sequence, target-conditioned view selection, and an extra 2D VLM call; LensDistill removes that inference cost by moving the detail into the student's native inputs. For a reader deciding what to take from this work, the decision-relevant comparison is not captioning versus grounding in the abstract but whether the target-conditioned evidence is worth its cost during training, since after distillation the student needs neither the privileged views nor the external VLM call.

Sources used in this answer