How can applied multisensory insights from VR and spatialized audio be integrated with computational and neural models?

Why pair a headset with a model at all?

VR and spatial audio are useful to modelers because they make sensory input adjustable in ways the real world does not allow. You can hold the visual scene fixed and change only how sound is rendered, or add or remove a cue and watch what the person does. That control is what a computational model needs: a defined input, a measurable output, and a way to test whether the model's internal steps match human behavior.

The review by Bortolotti and colleagues makes this case directly, arguing that spatialized audio and virtual reality should be more systematically integrated with computational and neural models of multisensory integration [1]. The framing matters: the claim is about how research should be organized, not a report that a unified framework already exists.

What an actual VR-to-model pipeline looks like

One concrete example comes from posture research. Zanchi and colleagues combined visual, vestibular, proprioceptive, and auditory inputs in a mixed-reality headset with auditory stimuli, then used a neural mass model — a simplified mathematical description of how groups of neurons interact — to analyze the multisensory interactions [2]. The measured behavior was postural stability, and the modeling was used to connect that stability to modality-specific contributions rather than treating balance as one undifferentiated skill.

A second example comes from animal work. Si and colleagues built a VR-based spatial navigation task for rodents that combines visual, auditory, tactile, and gustatory cues, and they used an A* algorithm — a standard path-planning method — to generate optimal routes through the environment [4]. Here the computation is doing a different job: it defines what an optimal path would be, so that the animals' choices can be compared against a benchmark. The study frames its aim as clarifying the computational mechanisms and neural basis of multisensory strategies [4], which is a statement of intent rather than a finished result.

What the headset alone can tell you about sensory weighting

Not every VR study needs a neural model to produce a useful constraint. Potter and colleagues varied spatial audio rendering and video resolution independently in a virtual environment with music and speech, and measured perceived audio-visual quality and immersion [3]. Both factors had statistically significant effects on both outcomes. The striking result was a trade-off: adding room acoustic rendering to head-tracked binaural audio improved immersion by as much as raising video resolution five-fold, from 0.5 to 2.5 megapixels per eye, under the conditions tested [3].

That kind of result is exactly the sort of input a model of multisensory weighting would need, because it quantifies how much one sensory channel's fidelity is worth relative to another's. The scope matters, though: the comparison holds for the specific stimuli, rendering levels, and headset setup used in that experiment [3], so it is a calibration point rather than a universal exchange rate between sound quality and picture quality.

The technical side of spatial audio is also where the modeling gets its parameters. Raposo and colleagues describe binaural rendering and Higher-Order Ambisonics — methods for computing how sound arrives at each ear from different directions — and trace spatial hearing from simple vibration up to neural perception, in a context where multisensory integration and audio-haptic feedback shape the perceived scene [5]. That work supplies the signal-processing detail but does not itself build a computational or neural model [5], which is a fair picture of where much of the field sits.

Where the loop is still open

Put together, these studies show two working patterns rather than one settled method. In the first, a model is fitted to behavior measured under controlled multisensory VR input, as with the neural mass model of postural stability [2]. In the second, a computational algorithm defines a benchmark against which behavior is judged, as with optimal path planning in the rodent navigation task [4]. Neither pattern yet closes the circle by using model predictions to design the next VR stimulus and then testing those predictions.

The review's recommendation points at that gap: the argument is that VR and spatialized audio should be brought into computational and neural modeling more systematically [1], which implies the current state is a collection of adjacent efforts. The immersion findings [3] and the rendering techniques [5] give modelers something to calibrate against, while the posture and navigation studies [2][4] show what a combined pipeline can look like. What is not yet established from this evidence is a shared framework that turns VR measurements into model parameters and back again.

Sources used in this answer

Related research questions