Does large-scale human vision and touch data improve visual-tactile robot learning?

What the 500 hours actually contain

TouchScale is a 500-hour record of contact-rich human interaction, captured with a single unified wearable setup rather than pooled from different sensors. Each recording temporally aligns egocentric RGB-D video from the head with wrist RGB video from both hands and dense full-hand bimanual tactile measurements from gloves. The collection spans roughly 87K episodes and about 2K predefined task descriptions, more than 1.5K objects, 9 high-level collection settings and over 800 distinct scene configurations, with about 20 participants contributing.

The sensing detail matters for interpreting every later number. The gloves report 880 taxels per hand, which the paper describes as the densest glove-based tactile sensing among the compared datasets, and all recordings share the same generation of hardware and synchronization pipeline. Because the sensor and collection protocol stay fixed, the paper can vary only the amount of training data and attribute changes to scale rather than to a change of equipment.

Why scale and consistency must arrive together

The paper frames the gap precisely: egocentric video datasets now span hundreds to thousands of hours, but they record how hands move while leaving contact and pressure unrecorded. Existing visual-tactile datasets supply that missing signal, yet most contain fewer than 30 hours of recorded interaction, and the largest resources merge recordings from different sensors or annotation procedures. Pooling such sources introduces a second problem, since differences in sensing hardware and collection protocol confound comparisons across them.

TouchScale is built to remove that confound. With one wearable setup deployed repeatedly across participants and environments, and without requiring manual action annotation, the dataset offers a collection paradigm the authors argue can be scaled further. This is the design choice that makes the scaling question answerable at all, and it is why the paper's comparisons are framed as fixed-protocol ones.

Contact prediction transfers to an unseen sensor

The first test asks whether training on TouchScale enables zero-shot vision-to-touch prediction on a tactile sensor unseen during training. Contact IoU (cIoU) measures how well predicted contact regions overlap the true ones, so higher values mean the model localizes where contact occurs more accurately. Training on the full TouchScale raises zero-shot cIoU on data from an unseen tactile sensor from 0.134 to 0.383 compared with prior tactile data.

A size-matched comparison isolates whether the gain is just volume. With the same prediction architecture, a model trained on a 16-hour subset of TouchScale reaches 0.181 cIoU on EgoTactile, against 0.134 for one trained on the 16-hour EgoTouch training split. At equal hours, the TouchScale data already predicts contact better, and the full dataset extends that lead. The scaling check holds the sensor and protocol fixed: moving from one tenth of TouchScale to the full dataset raises zero-shot cIoU on EgoTactile from 0.311 to 0.383.

Tactile supervision also shapes visual representations

The second test asks whether tactile supervision on TouchScale learns visual representations that transfer beyond tactile prediction. Pretraining a visual encoder on TouchScale yields the highest action recognition accuracy on three benchmarks among the compared visual-tactile datasets, under both linear probing and end-to-end fine-tuning. This is a different question from contact prediction: here the measure is recognition accuracy on action benchmarks, not overlap of predicted contact regions.

The result indicates that the visual features shaped by touch supervision carry information useful for recognizing what hands are doing, not only where they touch. Because the comparison is against other visual-tactile datasets under the same two evaluation protocols, the advantage is attributed to the pretraining data rather than to a different downstream setup.

Mid-training lifts robot success without action labels

The third test moves from perception to control. TouchScale is used only for visual-tactile mid-training of the N0-VTLA policy, without retargeting human hand motion to the robot and without human action labels. During this stage the vision-language-action backbone is frozen and only the tactile projection layer, the tactile predictor and an auxiliary tactile reconstruction head are updated, about 123.8M parameters while the remaining 3.71B stay frozen. The predictor is trained to represent the expected tactile change over a future horizon of 50 frames.

After mid-training, the checkpoint is post-trained on robot demonstrations. The baseline starts from the same released checkpoint but skips TouchScale mid-training, and both variants use the same 50 demonstrations per task and the same optimization and data-processing protocol, so the only difference is whether the tactile predictor received action-free human visual-tactile mid-training. On that controlled comparison, the average real-world success rate across four contact-rich manipulation tasks rises from 22.5% to 57.5%.

More data keeps helping under a fixed protocol

The fourth test asks whether gains grow with data scale when the sensor and collection protocol stay fixed. Growing the mid-training data from one fifth to the full dataset lifts robot success from 30.0% to 57.5%, and the zero-shot cIoU curve rises from 0.311 to 0.383 over the corresponding range. The paper describes both as an overall upward trend rather than a smooth monotonic law, and the tested range is what the evidence covers.

These two curves answer different questions and should not be merged: one tracks contact-region overlap on an unseen sensor, the other tracks task success on a physical robot. What they share is the experimental condition that makes them interpretable, namely that only the amount of TouchScale training data changed. The authors read the pattern as evidence that the scaling benefits reported for egocentric video also hold for visual-tactile learning.

Where the robot evidence stops

The robot result is bounded by its evaluation setup. The paper states that current robot evaluation is limited to one platform and four manipulation tasks, and that TouchScale contains no action labels. Broader evaluation across robot embodiments and contact-rich tasks is left as future work, so the 22.5%-to-57.5% figure should be read as evidence for this policy, these four tasks and this demonstration budget rather than as a general robot-learning claim.

The mechanism is also not fully settled. The authors report only a weak association between future-tactile and action prediction errors, and they describe the role of future-tactile prediction in robot control as an open question. The mid-training objective is therefore best treated as an empirically effective recipe whose causal path into control is not yet pinned down.

The comparison that should drive data choices

For readers weighing visual-tactile data sources, the most decision-relevant evidence is the fixed-protocol comparison rather than the headline jump. At matched 16 hours, TouchScale data reaches 0.181 cIoU against 0.134 for the prior split, which says the advantage is not only volume. The scaling curves then say that adding more of the same consistently sensed data keeps improving both contact prediction and robot success across the tested range.

The priority the paper establishes is therefore consistency first, then scale: a unified wearable pipeline that keeps sensing and synchronization fixed is what makes additional hours count, and the demonstrated payoff runs from zero-shot contact prediction on an unseen sensor through action recognition to real-world manipulation. The open questions are how far that payoff extends beyond one robot platform and four tasks, and how future-tactile prediction actually enters control.

Sources used in this answer