Learning to See While Learning to Act: Diffusion Models for Active Perception in Robot Imitation

2026-06-22

AI Summary

This work addresses the challenge of coordinating perception and action in partially observable environments where occlusions impede task execution. The authors propose See2Act, a novel approach that, for the first time, integrates diffusion models into the joint learning of active perception and action policies. By coupling action denoising with viewpoint optimization, See2Act predicts actions at test time through an actively inferred sequence of viewpoints. Crucially, it implicitly learns where to look and how to act from key-frame camera poses in offline demonstrations, eliminating the need for explicit viewpoint annotations. Experiments demonstrate that See2Act effectively recovers informative viewpoints in Ravens, achieves up to a 34% performance gain on RLBench tasks, and enables zero-shot transfer to real-world occluded grasping scenarios using only 50 digital twin demonstrations.

Abstract

Most imitation learning methods assume full observability in table-top settings. In practice, objects are often occluded, requiring robots to both search and act, and learning this coupled behavior from limited demonstrations remains challenging. We propose See2Act, an imitation learning approach that conditions action prediction on a sequence of actively-inferred viewpoints at test time, by coupling action denoising with viewpoint refinement. The policy is trained using camera poses anchored to keyframe actions from offline demonstrations, enabling implicit learning of where to see, while learning how to act. We empirically demonstrate that in Ravens the policy recovers informative viewpoints under severe occlusions, and on RLBench tasks it improves performance by up to 34% over prior methods. In the real world, we collect 50 demonstrations in a digital twin and achieve zero-shot sim-to-real transfer on pick-and-place tasks using depth observations. The policy handles significant occlusions, showing that learned viewpoint reasoning enables robust manipulation under partial observability.

Problem

• active perception
• imitation learning
• partial observability
• occlusion
• viewpoint reasoning

Innovation

• active perception
• diffusion models
• imitation learning
• viewpoint reasoning
• sim-to-real transfer