Object graphs target interaction failures that images and joint signals miss
The paper starts from a practical gap. Learned robotic policies offer no guarantees beyond the distribution of their training demonstrations, so a monitor must flag deviations at runtime without ever seeing a failure example. Existing monitors split into two families: visual detectors that compare raw images or learned visual features, and kinematic or action-based monitors that compare trajectories or joint signals. Visual detectors react to benign background, lighting and clutter changes; kinematic monitors verify only that the motion looks familiar and cannot tell whether the robot is touching the wrong object.
The authors argue that a large class of manipulation failures is neither visual nor kinematic: grasping the wrong item, violating task order, or placing an object in the wrong spatial relation to others. These errors can produce nominal motion and only subtle pixel changes. Their fix is to recast failure detection as monitoring how object interactions evolve, using a graph whose nodes encode object states and whose edges encode pairwise spatial relationships over time. In that framing, visual detectors act on node appearance and kinematic monitors track motion along trajectories, while the graph adds the relational structure needed for interaction-level errors.
Two scores read the same graph in different ways
On top of this representation the paper studies two complementary failure scores. Trajectory Projection and Correlation (TPC) is non-parametric: it compares each tracked object's trajectory with a set of nominal expert trajectories using an orthogonal point-to-polyline distance, and separately compares the pairwise Pearson correlation structure of object motions against the expert correlation matrix. The combined score is the sum of the projection term and the relational penalty, so it can be run as projection only, correlation only, or both.
GnnT is a learned alternative. A two-layer edge-aware Graph Attention Network encodes intra-frame geometry, a four-layer Transformer encoder models temporal dependencies over an observation window of 20 frames, and a shared MLP regresses each object's future position over a 15-step horizon. The hypothesis is that nominal manipulation obeys predictable physical and interaction rules, so a failure will violate the learned spatio-temporal priors and appear as a spike in prediction error. Both scores are calibrated with a conformal band built from nominal trajectories, and no demonstration is used both to fit the detector and to calibrate its threshold.
TPC leads on average AUPR and MCC with low seed variance
The headline comparison is between the two scores and image-based detectors. Both object-centric scores outperform image-based failure detectors, particularly on semantic and interaction-level failures. TPC provides the strongest performance in AUPR and MCC on both dataset averages, with low variance across seeds, and the discussion adds that it requires no training and the lowest inference cost. GnnT only approaches TPC in the continuously dynamic conveyor task, which the authors read as the setting where learned dynamics may justify their extra complexity.
The evaluation covers a public benchmark and a newly collected dataset of semantically complex manipulation tasks, across six tasks and ten random seeds. Metrics are reported as mean plus or minus standard deviation over seeds, so the low-variance claim is about seed-to-seed stability, not about performance on unseen task families.
Projection and correlation matter differently by task
The TPC ablation separates geometric from relational cues. On Waste-sorting, orthogonal projection alone achieved the best TPR, AUPR and MCC, suggesting failures there are mostly deviations from nominal trajectories. On Table-setting, correlation alone reached the highest TPR (94.9%), indicating many failures come from incorrect coordination between objects even when local motion looks plausible. Combining both gave the best TNR (97.1%), AUPR (96.3%) and MCC (88.8%) on Table-setting, 7.5 points above correlation only and 10.6 points above projection only, while keeping TPR at 92.3% versus 94.9%.
On Waste-sorting the combined score shifted to a conservative operating point: the highest TNR (96.8%) but a much lower TPR (47.7%), and MCC of 55.1% versus 61.0% for projection only. Because its AUPR stayed close to projection only (65.8% versus 67.7%), the authors argue the ranking ability is largely preserved and the sensitivity drop mainly reflects the decision threshold rather than lost discriminative power. The practical reading is that the relational term adds little in simpler, spatially dominated tasks and should not be assumed to help uniformly.
Removing object edges hurts most on relational tasks
For GnnT, removing the edge model degraded performance on both tasks, but the size of the drop depended on relational complexity. On Table-setting the loss was severe on threshold-independent and balanced metrics: AUPR fell 20.7 points and MCC 11.2 points, with 5.6 points more false alarms in TNR, showing the model could not reliably separate benign variation from relational failures without explicit interactions. On Waste-sorting the effect concentrated on sensitivity, with TPR down 8.1 points and MCC down 4.1 points while TNR and AUPR stayed comparable.
Removing edges also made predictions less stable across seeds, for example a TNR standard deviation of 8.3 versus 3.5 on Waste-sorting, which the authors interpret as relational structure acting as a regularizer. The spatio-temporal ablation points the same way: the naive MLP baseline had the lowest AUPR and MCC on both tasks, and its high TPR on Table-setting (88.7%) came with a very low TNR (46.2%), meaning it flagged many nominal episodes as failures. The full GnnT had the highest MCC on both tasks and the highest AUPR on Table-setting.
Graph quality depends on the upstream tracking pipeline
Because the graph is built from tracked objects, the detector inherits the tracker's errors. The pipeline first uses a vision-language model for open-vocabulary detection and initial bounding boxes; on 341 episodes the base Qwen model reached a 92.4% correct detection rate, with 5.3% missing objects, 1.8% identity confusion and 0.6% localization inaccuracies. A standardization loop that re-prompts when the expected object count is not found raised overall success to 94.4% and cut the missing rate by 2.1%.
Tracking itself was more reliable when initialized with ground-truth boxes: SAM2 averaged 98.3% perfect tracking, 0.2% recovered, 1.0% missing and 0.6% fusion across the datasets. The authors nonetheless list severe occlusions, missed detections and identity switches as factors that degrade graph quality and propagate to the anomaly detector. The representation also assumes discrete objects, so highly deformable materials, fluids and domains without clear object decomposition fall outside its scope, and modelling object centroids only means orientation and fine-grained pose failures cannot be captured.
Sources used in this answer
Robot-WATCHDOG : failure detection through object-centric graph representation
TPC, a training-free object-centric score combining trajectory projection and inter-object correlation, achieved the best average AUPR and MCC on both datasets with low seed variance, and both object-centric scores beat image-based detectors especially on semantic and interaction-level failures.