Can fMRI and camera motion reconstruct 3D room layouts without images?

Recovering wall distances without any video frames

Most fMRI decoders that produce 3D output reconstruct a picture or an isolated object. This study asks a different question: can brain responses to indoor walkthrough videos, combined with the known path the camera took, recover the metric geometry of a room the model has never seen? Metric matters here. Walls have to sit at the right distances, and views from different positions have to describe the same space.

The authors argue this is hard for three reasons. The viewpoint changes every second, the blood-oxygen-level-dependent (BOLD) signal that fMRI measures blurs several seconds together, and recordings are few. So the method does not try to decode depth maps directly. Instead it decodes a small layout code of 22 numbers per second describing wall distances, floor extent, openings and depth range, and later turns that code into depth and 3D surfaces.

The paper presents this as the first task of its kind: reconstructing 3D layouts of held-out scenes from camera trajectories and fMRI signals alone, with no visual frames as input. The claim is about a new task and a first working method, not about a solved problem.

Layout r scores depth correlation, not surface accuracy

The headline number is layout r, defined as the Pearson correlation between predicted and ray-cast log depths. Ray casting is the geometric reference: from a known camera pose, the true room geometry determines the depth of every pixel. Layout r therefore asks how well the predicted depth ordering and spacing match that reference, aggregated over the scene.

This is not the same as surface accuracy. The paper reports separate surface measures, including F-score at a 25 cm threshold, precision, recall and Chamfer distance in metres, computed on merged 3D surfaces. Those measures behave differently from layout r, and the paper is explicit that surface scores separate methods far less than layout scores do.

Keeping the two apart is essential. A method can improve layout r while leaving surface accuracy unchanged, and the paper reports exactly that pattern for its own layout component.

Fusing a neural layout code with camera free space

DRAFT has two inputs. The first is a compact layout code read out from early visual cortex (EVC) with ridge regressions, using 16 principal-component scores as features plus 12 camera-motion regressors. EVC responses map the image point by point, so they carry image structure that correlates with depth. The second input is the camera path itself: every visited position is free space, so the trajectory shows where walls are not. The authors use it as a learned prior, not as a hard bound on wall distance.

The fusion is deliberately coarse-then-fine. The code is rendered as coarse depth, a small generator trained on real readout errors adds detail to eight consecutive views, and DRAFT then replaces the broad low-spatial-frequency structure of those views with a 3×3 layout predicted from the code and the camera path. The views are placed in 3D at the known poses.

Because the readout also uses camera-motion regressors, the fusion gain alone does not isolate the brain. The authors add two controls that keep the motion regressors: replacing the EVC features with their training-scene mean, and circularly shifting each recording's BOLD series by 37 of 80 seconds. Both lower layout r and F-score in all ten scenes.

The camera path supplies most of the layout

From the camera path alone, the trajectory head reaches a layout r of 0.446. DRAFT reaches 0.480. The gap is real but modest, and the paper states plainly that most of the layout comes from the camera path.

The brain's contribution is measured in several ways. Adding EVC to the 12 camera-motion regressors raises layout r from 0.338 to 0.440 in nine scenes. Fusing the code with the trajectory layout raises layout r by 0.035 in all ten scenes and lowers pixel MAE in nine. Replacing EVC features lowers layout r by 0.037 and F-score by 0.036; shifting the BOLD series lowers them by 0.043 and 0.044. At the readout level, the EVC gain peaks at a lag of 3 to 4 seconds and beats all 20 circular shifts.

Among the tested regions, EVC predicts the code best. With 1,000 voxels per mask plus the motion regressors, EVC beats motion alone in all ten scenes and beats the scene-selective regions OPA, PPA and RSC together in nine. The authors caution that without functional localizers or dorsal motion areas this only ranks the masks and shows no biological cause.

DRAFT outperforms seven adapted fMRI decoders

On ten held-out scenes with 20 participants, DRAFT's layout r of 0.480 exceeds the best of seven adapted fMRI decoders at 0.429, the fixed template at 0.324, and the generator alone at 0.303. The comparison is not purely architectural: unlike those decoders, the fusion head is fitted to the same cell depths that layout r scores, and given the camera path through the trajectory layout each decoder still trails DRAFT by 0.024 to 0.032 across ten-scene means.

The gain is more than a generic guess about indoor rooms. Indoor views share a shape, floor near and walls farther away, so a template gets part of the layout right. DRAFT beats the fixed template in nine scenes and the per-second median in all ten, while Brain-Diffuser, MindEye and the generator alone do not beat the fixed template.

DRAFT also works as a plug-in without retraining. Applied to all seven decoders, it raises their layout r from a range of 0.171 to 0.429 up to 0.487 to 0.493. For Brain-Diffuser this lifts layout r from 0.324 to 0.493 in all ten scenes, the highest fMRI-based value reported. The authors note that layout r then mostly scores the DRAFT layout itself, so surfaces become the real test.

Surface accuracy separates methods far less

On merged 3D surfaces the picture narrows. After range calibration, DRAFT has a lower Chamfer distance than the generator alone in eight scenes, higher precision in all ten, and a similar F-score. But its Chamfer distance is not lower than the generator's at the generator's own selected factor of 1.5. Unmodified MindEye and Brain-Diffuser reach slightly higher mean F-scores than DRAFT, 0.494 and 0.492 against 0.484, and lower Chamfer distances, though only MindEye's lower Chamfer distance holds beyond noise.

Two properties of the surface scores explain the compression. First, even a wrong scene scores well: depth predicted for another scene and lifted at the evaluated poses keeps most of the F-score, and the correct scene adds only 0.055 to 0.069. Second, the scores reward depth range, and decoded images from a VAE reach 0.84 of the true spread against 0.46 to 0.54 for the ridge layouts, which raises recall even for a wrong layout.

Once depth scale is removed, DRAFT has lower scale-invariant depth error than the trajectory head, the generator, Brain-Diffuser and MindEye in all ten scenes. The authors call this a low bar, since Brain-Diffuser and MindEye still have higher scale-invariant error than the fixed template.

The brain gain replicates in only two withheld scenes

The paper's own robustness check is the most useful guide to how far the result travels. In five withheld scenes, DRAFT's absolute layout r of 0.489, 0.476, 0.579 and 0.534 in four of them fell inside the prediction interval from the ten development scenes. The advantage over the fixed template replicated in all five. The advantage over the trajectory head replicated only in the first two, and the advantage over feature replacement replicated in three of five.

Pooling matters too. On single recordings without pooling, DRAFT beat the trajectory head in 31 of 56 recordings in the first scene but only 10 of 24, 2 of 6, 10 of 24 and 11 of 24 in the later scenes. The gain over the path appears when recordings are pooled across participants.

The authors list the boundaries directly: DRAFT needs known camera paths and pooled recordings, was developed on these scenes with several analyses post hoc, and its brain gain was not measurable beyond low-level image features. The method has not been tested for unconstrained or covert decoding.

A coarse layout prior, not a finished room reconstruction

The most defensible reading is that camera motion already carries most of the recoverable room geometry, and fMRI adds a small, consistent increment on top of it. The increment is visible in every development scene and survives two brain-input controls, but it replicated in only two of five withheld scenes and only when recordings were pooled.

That ordering changes how the numbers should be compared. A layout r near 0.48 is a coarse geometric correlation, not a surface reconstruction: the same paper shows surface scores barely separating DRAFT from decoders it clearly beats on layout r, and shows a wrong-scene depth map retaining most of the surface F-score.

The practical priority the paper establishes is therefore the plug-in role. The DRAFT layout improves all seven decoders without retraining and gives Brain-Diffuser and MindEye the best fMRI-based mean surface scores, with Brain-Diffuser at 0.499. For readers weighing whether brain signals add anything beyond a known trajectory, the answer is a small yes under pooled, known-path conditions, and the paper's own replication record is the threshold to watch.

Sources used in this answer