How does visuo-tactile for deformable object manipulation compare with retrieval, fine-tuning, and human review?

What has changed: Visuo-tactile is now the proven winner over vision-only for precise manipulation

For years, robot manipulation relied almost entirely on cameras. But recent work shows that adding tactile (touch) sensing to vision—called visuo-tactile feedback—dramatically improves performance on tasks that require millimeter-level precision, like inserting a plug or packing objects. A 2025 study on terminal assembly (connecting electrical terminals) found that a visuo-tactile policy achieved 100% successful insertions across 100 different starting positions, while a vision-only imitation learning policy succeeded only 9% of the time, and an online reinforcement learning (RL) policy failed completely (0%) [1]. This is a stark reversal: touch is no longer a nice-to-have; it's essential for tasks where objects are fragile or deformable.

Similarly, a 2023 study on USB insertion—a classic industrial task—showed that a visuo-tactile policy succeeded 45 out of 45 times across different grasp poses, whereas a behavior cloning agent trained on 50 human demonstrations succeeded only 1 out of 45 times, and an online RL policy (TD3) succeeded 0 out of 45 [5]. The message is consistent: vision alone is not enough when the robot must feel its way into a tight fit.

How retrieval, fine-tuning, and human review fit into the new picture

Retrieval—using keypoint correspondences from visuo-tactile images to guide actions—offers a lighter-weight alternative to full policy learning. A 2024 study showed that by extracting keypoint correspondences from visuo-tactile sensor images, robots could perform precise tasks like block alignment and gear insertion with millimeter-level accuracy, without the need for extensive training or post-grasp adjustments [4]. This is especially useful when object locations are not predefined, because the robot can 'retrieve' the right action by matching tactile features to visual features. The trade-off is that this method may not generalize to highly variable tasks as well as learned policies, but it excels in deployment efficiency.

Fine-tuning and human review are not separate from visuo-tactile learning—they are often built into the training process. For example, the 2025 terminal assembly study used human demonstrations and interventions during training to ensure safety, and the 2023 USB insertion study used force-torque sensing to create a safe self-supervised data collection pipeline, reducing the need for human input [1][5]. This suggests that while visuo-tactile policies can learn from scratch, they benefit from a human-in-the-loop to avoid damaging parts during exploration. In contrast, the 2023 packing study used a deep reinforcement learning (DRL) pipeline that learned to select optimal actions from visuo-tactile feedback, achieving better accuracy and efficiency than existing methods [2]. So, the answer is not 'either/or'—the best systems combine visuo-tactile sensing with retrieval, fine-tuning, and human oversight to balance precision, safety, and adaptability.

When visuo-tactile works best—and where it still struggles

Visuo-tactile feedback shines in tasks that involve tight tolerances, fragile objects, or long-horizon manipulations. The 2024 3D-ViTac system demonstrated that even low-cost robots can perform precise manipulations and significantly outperform vision-only policies, particularly in safe interactions with fragile items and long-horizon tasks involving in-hand manipulation [3]. The key is that tactile sensors provide dense contact information that vision cannot capture, especially when objects are occluded or deformable.

However, the evidence also shows that visuo-tactile policies are not a silver bullet. They require careful sensor integration and often need human demonstrations or interventions to avoid catastrophic failures during training [1][5]. Moreover, the success rates drop dramatically when the policy is not trained with visuo-tactile data—as seen in the 9% and 0% baselines [1]. So, while the technology is powerful, it demands a significant investment in hardware and training infrastructure. For tasks where objects are rigid and positions are fixed, simpler vision-based methods might still suffice, but for deformable or fragile objects, visuo-tactile is the clear winner.

About These Sources

This answer is built on 5 peer-reviewed studies — published from 2023 to 2025, 3 from 2024 or later, 2 in Q1–Q2 journals — selected as the most relevant from 5 studies that passed quality screening, drawn from 33 papers retrieved from a database of over 500 million.

Sources used in this answer

Related research questions