Can task-ready simulation assets be built from single images by retrieval?

What counts as a physically valid asset

A simulator cannot use a mesh that merely looks like a cabinet. Everyday objects are articulated: cabinet doors rotate about hinges, drawers translate along rails, and storage bins slide within frames. To reproduce that behavior, the simulator needs multiple rigid bodies connected by joints, with collision geometry and physical parameters that stay stable under contact.

AffordCraft's pass criterion follows from this. An input passes only if the method exports a structured asset and that asset then clears a physical gate in simulation; an input with no structured export counts as a failure, so nothing is quietly dropped from the denominator. The headline rate is therefore passes over all inputs of the fixed evaluation set, not over the subset that produced an export.

This distinction matters because export and physical validity are separate hurdles. In the paper's comparison table, PhysX-Anything exports a structured asset for 97.85% of inputs but passes the physical gate as delivered for 44.55%; PhysX-Omni exports 98.05% and passes 36.50%. A high export rate does not imply a usable asset.

Retrieval turns reconstruction into matching

The paper's premise is that articulation structures are highly repetitive. A drawer may slide along the same axis in both a desk and a filing cabinet. If a mechanism is modeled and physically verified once, it can be reused across many objects, which removes the need to recover parts and joints anew for every photograph.

AffordCraft therefore treats single-view reconstruction as a constrained matching problem over known mechanisms. Given one RGB image and a task instruction, it locates the object and the part the task acts on, determines the required motion such as rotation about a hinge, retrieves library entries by category and visual similarity, and has a multimodal model check which entries provide that motion at the right part, or abstain if none does. It then scales and places the chosen entry without changing its parts or joints, and accepts the result only if it passes the physical test.

The output is a functional substitute, not a copy. The authors state that a cabinet of another color is usable as long as its door hinges where the task requires. They also argue that assets sharing a mechanism but differing in appearance give policy training appearance variation that a single exact copy would not.

How much more reliable than generation

On the single-object benchmark of 2,000 RGB photographs from 31 categories, drawn from a quality-screened subset of Open Images, AffordCraft passes the physical gate for 1,703 inputs, or 85.15%. The paper reports a Wilson 95% interval of 83.52–86.64% for that headline rate.

The five generative methods compared on the same photographs pass on at most 45%. The paper's table lists native pass rates of 44.55% for PhysX-Anything, 36.50% for PhysX-Omni, 23.70% for PAct, and no native pass for PartCrafter or TRELLIS.2, which are scored only after a common adaptation that adds collision geometry and mass properties.

Cost separates the methods further. At the median, the generative methods need 10 to 78 times AffordCraft's GPU time per valid asset. AffordCraft's own median time is 56.60 seconds per input with 53.1 GPU minutes per pass and 0.8 GiB peak memory, measured on a 32-input timing subset with one RTX 5090 per worker and the library's collision geometry built once.

Two non-generative baselines are also reported, but on a smaller 200-input set: Articulate-Anything passes 35.00% after adaptation, and a GPT-6 Astra agent passes 34.50% natively at a median of 1,423.0 seconds and 2.90 USD of model calls per input. Because these two run on 200 inputs rather than 2,000, their rates are not pooled with the main comparison.

Where construction fails by category

The 297 inputs without an asset break down by terminal reason: 230 are not localized, 64 exhaust the candidate budget, and 3 end with an invalid automatic box or a non-object reply. Across 5,874 candidate attempts, 1,703 were selected, 2,573 produced no export, 1,093 were semantically rejected, 435 failed the physical gate, and 70 were repaired but still produced no export.

Failures concentrate in specific categories. Window misses 106 of 263 inputs, Chair 42 of 153, Table 33 of 264, and Bottle 32 of 159. At the other end, Door passes 120 of 145 and StorageFurniture 137 of 140. Several small categories pass at 100%, including Mouse at 59 of 59, Kettle at 23 of 23, Coffee maker at 19 of 19, Printer at 19 of 19, Trash can at 17 of 17, Microwave at 5 of 5, Remote at 5 of 5, Dishwasher at 1 of 1, and Scissors at 1 of 1, though the paper notes that small categories carry wide intervals.

A structure-preserving repair was applied to 179 of the 5,874 candidate attempts, and 60 passes use a repaired asset. The paper also records a limit of the source library: the driven joints of the printer and the kettle have millimeter ranges, so their two states coincide, and the bottle is a rigid entry with no joint.

Cluttered images and library growth

On 50 cluttered COCO images containing 237 annotated objects, the pipeline first detects objects and the annotations serve only for scoring. After automatic detection, 162 of 237 annotated objects pass the same physical test. The paper notes that objects in cluttered images are evaluated individually, without assessing scene depth, support relations, or layout.

Library size is a separate lever. Growing the library from 141 to 11,372 entries requires no change to the method. Across the five libraries, category coverage rises from 46% to 100%, and the share of selections carrying the requested label rises from 18% to 51%. The paper reports that coverage rises only when new categories arrive, while top-1 agreement rises at every step.

These two numbers answer different questions. Coverage asks whether a category is represented in the library at all; top-1 agreement asks whether the selected entry carries the requested label. The paper states that both compare labels, not whether the selected asset has the right part and motion for the task.

Which components change the verdict

The one-factor study is a separate campaign of the frozen method on the same registered 200 inputs with the same gate, with the frozen method rerun inside the study as the paired reference of every row. That reference passes 169 of 200, while the campaign recount on the same inputs passes 167 of 200; the paper reports them as separate runs.

Removing installation evidence changes no verdict at all across the 548 Window, Door, and Storage-Furniture inputs the two installation-related factors were written for. The frozen method passes 411 of those inputs. Restricting the decomposition cascade to level 0 loses three doors and nothing else, which the authors place on the thin frames the cascade was added for.

The paper is explicit that small differences in the ablations are not enough to establish a reliable improvement. Under the target-assisted development protocol, which supplies the registered target region and category label with the image, earlier configurations show scale handling mattering more: adapting the highest-ranked candidate passes 685, a calibrated scale estimate 1,043, and three scale hypotheses committed before any simulator feedback 1,172. Selection alone over a pool of five passes 887, and adding the articulation adapter's structure-preserving repair recovers 316 of the 558 cases with an invalid articulation, for 1,203. The paper states that the two protocols pose different problems and never pools their rates.

What the constructed assets support

The paper builds manipulation tasks from the constructed assets, both with single objects and in composed scenes. Policies trained on scripted demonstrations complete both kinds of tasks from initial states unseen in training. In the ten scenes, the scripted teacher completes 273 of 320 episodes.

The policy setup is specific. The backbone is OpenVLA-OFT with a LIBERO-Spatial checkpoint, frozen and never finetuned; the untrained condition uses the same action head at initialization. The head is a 512-unit GRU over 8,192-dimensional features from two 320×240 images, with 9 proprioceptive joint positions, an action chunk of 8 future targets with 4 executed, control at 10 Hz and physics at 60 Hz, trained with AdamW for 24,000 steps at batch size 16.

The paper's own limitation is that the manipulation tasks use a fixed set of assets seen during development, so generalization to unseen assets and transfer to real robots remain to be tested. It also notes that the single-object evaluation measures construction when the object category is given in the instruction rather than open-vocabulary recognition.

What decides usefulness here

The most decision-relevant split in this paper is not generation versus retrieval in the abstract but what each approach buys per unit of compute. AffordCraft's 85.15% pass rate on 2,000 photographs, at a median 56.60 seconds per input and 53.1 GPU minutes per pass, sits against generative methods that pass at most 45% and need 10 to 78 times the GPU time per valid asset at the median. For a reader deciding how to populate a simulator, the paper's evidence points to the library as the reusable asset and the per-image cost as the thing being amortized.

The second split is library size against selection quality. Going from 141 to 11,372 entries raises category coverage from 46% to 100% but top-1 agreement only from 18% to 51%, and the paper is careful that these compare labels rather than whether the chosen asset has the right part and motion. A larger library removes the coverage excuse without solving selection.

The third split is category. Door at 120 of 145 and StorageFurniture at 137 of 140 behave very differently from Window at 157 of 263 and Chair at 111 of 153, and the 230 not-localized failures out of 297 missing assets point at localization rather than at the physical gate. The paper's own priority is therefore the one it states: structure is modeled and checked once in the library and reused by every query, which is what makes the per-image cost and the per-image failure rate the quantities worth watching as the library grows.

Sources used in this answer