DF-ExpEnse: Diffusion Filtered Exploration for Sample Efficient Finetuning

2026-06-17

AI Summary

This work addresses the low sample efficiency in online fine-tuning of pretrained generative control policies, which stems from insufficient exploration quality. To this end, the authors propose a diffusion-based structured exploration method that leverages a diffusion filtering mechanism to construct a high-quality, evaluable set of multimodal action candidates. An ensemble of critics is employed to select actions that balance performance and exploration value by exploiting their predictive uncertainty. Furthermore, the approach introduces a cross-agent collaborative exploration mechanism for multi-agent settings. This study represents the first application of diffusion models’ multimodal modeling capability to action exploration in reinforcement learning fine-tuning. Empirical results across diverse manipulation and locomotion tasks demonstrate substantial improvements over baseline strategies, confirming the method’s generality and efficiency.

Abstract

A natural recipe for intelligent robotic decision-making is initializing from pretrained generative control policies, which have summarized offline experience, and adapting them to self-collected online experience. We present DF-ExpEnse, an exploration technique that improves the quality of online experience collection, thus increasing finetuning sample-efficiency. DF-ExpEnse leverages the multimodal modeling capabilities of the generative control policy to create an expressive and tractably evaluatable candidate set. It then utilizes an ensemble of critics to identify the action that best balances quality with high exploration interest. In fleet settings, DF-ExpEnse further enables cross-agent communication to facilitate collaborative exploration as a group. DF-ExpEnse can be seamlessly integrated with existing strategies that finetune pretrained generative control policies via reinforcement learning. We experimentally validate consistent sample-efficiency benefits through DF-ExpEnse across a variety of manipulation and locomotion tasks, compared to default finetuning and alternative action selection schemes. Project can be found at https://df-expense.github.io.

Problem

• sample efficiency
• finetuning
• exploration
• generative control policy
• robotic decision-making

Innovation

• Diffusion Filtered Exploration
• Generative Control Policy
• Sample-Efficient Finetuning
• Ensemble of Critics
• Collaborative Exploration