Can binary task rewards alone teach a robot to assemble NIST task board parts?

How much assembly strategy must be specified

Assembly Task Board #1 contains 20 assembly pairs, each a movable part plus its fixed mating geometry on the board. The tasks include peg insertion, gear meshing, electrical connector insertion, and nut threading. Parts are as small as 4 mm in diameter, clearances between mating surfaces can be as low as zero, and a nut can tilt, cross-thread, or jam instead of advancing along a bolt.

Prior learning-based work has generally narrowed this problem: fixing the assembly sequence, constraining initial part poses, assuming accurate pose estimates, training one specialist policy per task, or covering only a subset of the board. Demonstrations, task decomposition, and dense progress rewards all supply guidance about how to assemble. The paper asks how much of that guidance is actually necessary.

The reference point is the AlphaGo to AlphaGo Zero progression, where learning from a terminal win/loss objective supported strategy discovery without human game data or intermediate reward shaping. AlphaNIST tests the analogous claim in continuous, contact-rich manipulation: can a binary success signal plus task diversity let one policy discover assembly strategies on its own?

A reward that fires only on full assembly

Each task is modeled as a finite-horizon Markov decision process, and one shared policy acts on partial observations. Task identity is not given as a discrete label; variation across the 20 assemblies reaches the policy through surface point clouds of the movable part, the mating geometry, and the robot, together with proprioception and the last action.

The task reward is binary. An assembly-relative coordinate places the part in its mating frame, and success is declared when the distance between the part's current relative pose and the fully assembled pose falls below 0.001 m. That reward is issued once and the episode terminates. Intermediate progress earns no task reward.

A control cost is subtracted at every step to penalize large actions, rapid joint motion, and excessive torque, with one coefficient fixed across all tasks. The paper states that this term does not measure assembly progress; it biases learning toward smoother, lower-effort behavior. In multi-pair episodes the success reward stays at zero until every active pair is assembled, with no prescribed order and no reward for partial completion.

Diverse resets decide what the policy can practice

Training draws initial states from a cached reset bank adapted from OmniReset. Each entry stores an assembly identity together with a simulator state, so sampling an entry selects both the task and its starting configuration. The bank spans ungrasped workspace, pickup, transport, near-mating, and partially assembled configurations, and it enables GPU-parallel initialization. These reset families define regions of the training distribution rather than a prescribed sequence; the policy receives no stage identifier.

Which entry gets sampled is governed by beta-distributed success-guided sampling. Each entry keeps a rolling empirical success rate over its most recent episodes, and its sampling probability is proportional to the square root of that rate times one minus that rate, plus a small constant that keeps every entry sampleable. The kernel therefore favors entries with intermediate success rates, neither already solved nor hopeless.

The three components have separate jobs: the reset bank determines which states are available, beta sampling determines which states are sampled, and the binary reward determines what counts as success. The uniform-sampling ablation keeps the bank, policy, reward, and optimizer unchanged and only flattens the sampling probabilities.

One shared policy beats twenty specialists on hard parts

The heterogeneity comparison trains one policy across all 20 tasks and compares it with 20 separately trained policies, using matched per-run interaction budgets and otherwise identical settings at 64K environments. The shared policy reaches 98.2% training success, above 98% on every task. The separately trained sparse-reward policies average 81.4%.

The gap concentrates on the small precision assemblies. Separately trained policies have at most 0.3% success on M4 nut threading, 4 mm peg insertion, and rod insertion, and the paper attributes the obstacle to unreliable grasping of small parts from the table. The shared policy produces successful grasp-and-assembly rollouts on those same tasks without task-specific changes. The paper's interpretation is that shared experience helps with assemblies that are difficult to learn in isolation.

Cost also differs. At 64K parallel environments across 25K iterations, the shared sparse-reward policy reaches its 98.2% in 410 GPU-hours, while the 20 separate policies consume 16.1K total GPU-hours, 39.3 times more device-hours. The joint sparse configuration also converges faster than the mean of the separate policies.

Dense progress shaping does not show the same benefit

The reward ablation adds two dense terms to the binary reward plus control cost: one rewards end-effector proximity to the part centroid, the other rewards alignment with the target assembly orientation, both with coefficients of 0.01 matching OmniReset. These are common shaping terms in manipulation and assembly reinforcement learning.

At their final checkpoints, the 20 separately trained dense-reward policies reach a mean success rate of 78.3%, ranging from 45.0% on M4 threading to 93.6% on the medium gear. Joint dense training reaches 17.8% overall success at 25K iterations, with low completion across tasks. The paper offers a possible explanation rather than a demonstrated cause: dense shaping terms may favor different partial solutions across geometries, whereas the binary reward gives every task the same full-assembly objective.

This comparison is between the specific dense terms tested and the binary objective, not between sparse rewards and all possible shaping. The paper reports that the binary task objective outperforms the evaluated dense shaping across all tested scales.

Sparse rewards need parallel experience and beta sampling

With the sparse binary reward and beta sampling, evaluation success rises from 38.6% at 1K environments to 78.9% at 4K and 86.9% at 8K. With dense reward shaping, success stays at 11.8% to 12.0% across 1K to 4K environments. These evaluations use stochastic actions, a common frozen mixed-reset bank sampled uniformly within each task, and require full assembly within 15 seconds without a simultaneous safety failure.

Reset sampling matters increasingly with scale. At 256K environments, the sparse binary reward reaches 97.4% with beta sampling versus 84.3% with uniform sampling; the corresponding dense-reward results are 77.4% and 34.3%. The sparse objective's advantage is already visible at modest scale, and beta sampling adds further gains as parallel experience grows.

The scaling setup increases GPUs, environments, and rollout batch size together while holding the task distribution, reward, policy, and environment fixed. The paper's claim is therefore about this joint scaling recipe, not about environment count alone.

Grasp adaptation and planning-like behavior emerge unsupervised

The learned policy exhibits behaviors that no reward term specifies. For an 8 mm peg, a center grasp may succeed at pickup but leave too little shaft exposed for insertion; the policy drops the peg and regrasps it near one end. For an M12 nut, it rolls the nut from its edge onto a flat face before placement. For an M4 nut, repeated one-sided taps advance the nut along the thread. It also uses board contact to reorient a waterproof connector and surrounding geometry to pivot a medium gear into a horizontal pose over its mounting post.

Longer-horizon coordination appears in the multi-pair setting. Success rates fall gradually as active pairs increase, from 99% with one active pair to 84% with ten, and the reward is issued only when all active pairs sit in their assigned board slots. In one example, the policy brings a USB-A connector from a distant corner to a nearby working position, leaves it, assembles the 8 mm rod, then returns to insert the connector. In another, it makes small turns while threading an M12 nut to avoid touching a neighboring RJ45 connector.

The paper describes these as planning-like: actions whose utility is realized later, and adaptation of one assembly to the state of another, without an explicit planner, policy switching, or prescribed sequence. This is the authors' characterization of observed behavior in simulation, not a claim that the policy maintains an explicit plan.

Simulation success is the boundary to watch

All reported results are in simulation. The setup uses a simulated 7-DoF Franka Emika Panda with identified inertial and drivetrain parameters, an implicit impedance controller using the manufacturer's stiffness and damping profile, and hardware-matched joint-speed and torque limits. Those limits keep actions inside a hardware-realistic envelope, but the paper states they do not establish robustness to perception error, latency, or contact-model mismatch. Transfer and repeatability on the physical board remain to be evaluated.

The multi-pair results also stop short of the full board. Applying the 10-pair policy to successive groups of parts is offered as a possible route, but extending the framework to all 20 parts under a single terminal reward is left as a research goal.

The paper's distinctive priority is the combination rather than any single ingredient: one shared policy, a binary full-assembly reward, a diverse reset bank, beta success-guided sampling, and large-scale parallel training. Within that recipe, the most decision-relevant comparison is between the shared sparse-reward policy at 98.2% and the 20 separate sparse-reward policies at 81.4%, with the largest gaps on the small precision assemblies that specialists rarely solved. The dense shaping tested here did not reproduce that pattern, reaching 17.8% under joint training at 25K iterations.

Sources used in this answer