MemCorr-DP: Counterfactual Correspondence Conditioning for a Diffusion Policy Guided by a Reference

2026-09-06

AI Summary

为解决视觉运动策略在物体位置和视角同时变化时失效的问题，提出MemCorr-DP方法，通过将参考轨迹与当前场景的3D关系显式化来指导扩散策略。

Abstract

Behavior-cloned visuomotor policies can remain accurate near their training distribution yet fail when object position and camera viewpoint change together. A successful reference trajectory contains the geometry needed to transfer the same interaction, but the policy must align that geometry with the current scene and remain sensitive to it during denoising. To address these challenges, we present MemCorr-DP, a diffusion policy that lifts frozen RoMa v2 matches into explicit 3D relations between the current scene and the reference trajectory. A counterfactual paired objective assigns opposite behaviors the same physical state and noisy action while retaining reference-specific denoising targets. Mixed-condition fine-tuning then adapts the policy from ground-truth geometry to measured correspondence errors. Our strongest evaluation places the Door in the outermost position bands beyond the training support and changes the query camera by $\pm15^\circ$. Under this combined shift, MemCorr-DP achieves 96.67% closed-loop success, compared with 88.00% for a visual Transformer with the same action architecture. Objective ablations and reference interventions show that behavior responds to the selected reference, while matched controls favor the complete relation set over future motion or centroid geometry alone. These results support explicit 3D reference relations as a robust conditioning interface when spatial and viewpoint changes are compounded in the evaluated task.

Problem

• behavior-cloned visuomotor policies
• object position
• camera viewpoint
• interaction transfer
• scene alignment

Innovation

• Counterfactual Correspondence Conditioning
• Diffusion Policy
• Explicit 3D Relations
• Mixed-condition Fine-tuning