Auditing Instruction-Trajectory Mismatches in Multimodal Robot Demonstrations

2026-08-07

AI Summary

This work addresses the implicit misalignment between language instructions and behavioral trajectories in multimodal robot demonstration data by proposing MMPF, a training-free auditing framework. Treating each modality as an expert, MMPF estimates task label distributions through a combination of local neighborhood consistency and global prototype similarity, then fuses multimodal information via prediction entropy weighting to achieve high-accuracy mismatch detection and label correction. As the first method specifically designed to handle data errors where actions are correct but instructions are erroneous, MMPF achieves state-of-the-art performance in instruction-trajectory matching (ITM) detection and correction on both the LIBERO benchmark and real-world robot datasets. Experimental results demonstrate that MMPF significantly enhances downstream policy learning, with physical robot trials validating an effective trade-off between filtering and relabeling strategies.

Abstract

Robot demonstration datasets used to train vision-language-action policies can contain a subtle but harmful failure mode: trajectories that are behaviorally correct but paired with the wrong language instruction. We study post-hoc auditing of these Instruction-Trajectory Mismatches (ITMs). Unlike failed rollouts, ITMs often look plausible, and can corrupt the language-behavior mapping learned by the policy. We propose Multimodal Probabilistic Fusion (MMPF), a training-free auditing framework that treats each modality as an expert, estimates a task-label distribution from local neighborhood agreement and global prototype similarity, and then fuses modalities with predictive-entropy weighting in a product of experts. Across LIBERO benchmarks with injected instruction mismatches and noisy real-robot data, MMPF achieves the strongest overall ITM detection and label correction accuracy. We also show that auditing improves most downstream policy learning in settings where language is needed to disambiguate the task. We demonstrate in real robot experiments that our method can achieve improved policy performance and show the trade-off of filtering demonstrations compared to relabeling.

Problem

• Instruction-Trajectory Mismatches
• multimodal robot demonstrations
• vision-language-action policies
• language-behavior mapping
• robot demonstration datasets

Innovation

• Instruction-Trajectory Mismatch
• Multimodal Probabilistic Fusion
• Training-free Auditing
• Vision-Language-Action Policy
• Demonstration Relabeling