[CoRL 2024] How to Peel with a Knife: Aligning Fine-Grained Manipulation with Human Preference

TL;DR

Peeling a potato with a knife seems simple for a human, but for a robot, it is a nightmare of contact-rich dynamics and subjective quality standards. This paper introduces a two-stage framework that first learns a baseline peeling skill via force-aware imitation learning and then "polishes" the policy using a reward model trained on human preferences. The results? A robot that can peel cucumbers, apples, and potatoes with over 90% accuracy, even generalizing to fruits it has never seen before.

The "Quality" Bottleneck in Robotics

In traditional robotics, a task is often defined by a binary reward: did the arm move the block from A to B? However, in tasks like cooking, surgery, or craftsmanship, quality is continuous and subjective. Is the peel too thick? Is the cut smooth?

Existing methods fail because:

1. Data Quantity: Collecting force-sensitive data is slow and expensive.
2. Metric Mismatch: Mathematical metrics (like distance or torque) don't always align with what a human considers a "good" result.

The authors solve this by asking: Can we learn what a 'good peel' looks like from human feedback?

Methodology: The Two-Stage Alignment

The researchers break the problem into a "Learn then Refine" architecture.

1. Robust Initialization (Imitation Learning)

Using a 7-DoF Kinova Gen3 arm and a custom knife mount, the team collected 50–200 trajectories via SpaceMouse teleoperation.

• Force-Awareness: The policy consumes RGB-D vision, joint positions, and 6-axis force-torque sensor data.
• Diffusion Policy: They utilize a Diffusion Policy for its ability to handle multi-modal distributions and complex trajectories.

2. Preference-Based Refinement (The Secret Sauce)

After the base policy is trained, it might "succeed" but produce poor-quality results (e.g., cutting too deep).

• Hybrid Reward Model: The authors created a reward function combining Quantitative data (local peel thickness category) and Qualitative data (a 0-9 Likert scale of human preference).
• Residual Tuning: Instead of retraining the whole model, they froze the base policy and trained a Residual Policy that predicts "corrections" to the base actions to maximize the human-aligned reward.

Fig 1: The two-stage pipeline: from data collection to preference-based refinement.

Why SpaceMouse?

Interestingly, the authors compared different data collection methods. While VR teleoperation and kinesthetic teaching are common, they found SpaceMouse provided the best balance of precision and stability for the delicate task of holding a knife edge against a slippery surface.

Experimental Results: Precision & Generalization

The system was tested on cucumbers, apples, and potatoes.

• High Success: Reached 100% success on seen produce after refinement.
• The Power of Feedback: As shown in Table V, the "Preference-based" method significantly outperformed "Base Only" or "Quantitative Only" models.
• Zero-Shot Generalization: A policy trained only on apples could peel a pear with 90% success, despite the different geometry and skin texture.

Table 1: Human preference alignment results showing significant jumps in qualitative scores (Score A/P).

Critical Insights

1. Sensing Matters: Using two wrist cameras ("Before" and "After" views) was critical. The "Before" camera captured a less occluded view of the knife-skin contact point, providing vital geometric cues.
2. Force is Non-Negotiable: Ablation studies (Table III) showed that removing force-torque data dropped success rates to nearly 0% for some produce, proving that vision alone cannot handle the "unstable blade-surface contact" of peeling.
3. Residual over Scratch: Training the refinement from scratch failed completely. The "Residual Policy" approach acts like a human learning a basic skill and then fine-tuning their technique—a much more efficient learning paradigm.

Conclusion & Future Outlook

This work moves robotics away from "binary success" toward "human-aligned quality." While the reliance on manual teleoperation remains a limitation, the ability to achieve SOTA results with as few as 8 fruits (50 trajectories) is a massive win for data efficiency.

Future versions of this system might use Online RL to further reduce human intervention, or even "Sim-to-Real" transfer once cutting simulators become sufficiently high-fidelity to model the heterogeneous textures of organic vegetables.

Takeaway: If you want a robot to perform like a craftsman, you don't just need more data—you need to teach it what a craftsman values.