Sample-Efficient Real-World Dexterous Policy Fine-Tuning via Action-Chunked Critics and Normalizing Flows

2026-02-10

AI Summary

This work addresses the challenge of policy fine-tuning for real-world dexterous manipulation, where sample scarcity and highly multimodal action distributions hinder existing methods from achieving both efficient credit assignment and effective multimodal modeling. To overcome this, we propose the SOFT-FLOW framework, which—on a physical robot for the first time—integrates a normalizing flow–based multimodal generative policy with a chunked-action critic. This design enables precise likelihood estimation to support sequence-level value evaluation and introduces likelihood-based regularization for conservative policy updates. Evaluated on two challenging tasks—grasping tape with scissors and pronation-based cube rotation—the method significantly outperforms baseline approaches, achieving stable and sample-efficient policy adaptation.

Technology Category

• Intelligent Robots: Manipulation
• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision

Application Category

• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Real-world fine-tuning of dexterous manipulation policies remains challenging due to limited real-world interaction budgets and highly multimodal action distributions. Diffusion-based policies, while expressive, do not permit conservative likelihood-based updates during fine-tuning because action probabilities are intractable. In contrast, conventional Gaussian policies collapse under multimodality, particularly when actions are executed in chunks, and standard per-step critics fail to align with chunked execution, leading to poor credit assignment. We present SOFT-FLOW, a sample-efficient off-policy fine-tuning framework with normalizing flow (NF) to address these challenges. The normalizing flow policy yields exact likelihoods for multimodal action chunks, allowing conservative, stable policy updates through likelihood regularization and thereby improving sample efficiency. An action-chunked critic evaluates entire action sequences, aligning value estimation with the policy's temporal structure and improving long-horizon credit assignment. To our knowledge, this is the first demonstration of a likelihood-based, multimodal generative policy combined with chunk-level value learning on real robotic hardware. We evaluate SOFT-FLOW on two challenging dexterous manipulation tasks in the real world: cutting tape with scissors retrieved from a case, and in-hand cube rotation with a palm-down grasp -- both of which require precise, dexterous control over long horizons. On these tasks, SOFT-FLOW achieves stable, sample-efficient adaptation where standard methods struggle.

Problem

• dexterous manipulation
• sample-efficient fine-tuning
• multimodal action distributions
• credit assignment
• real-world robotics

Innovation

• normalizing flows
• action-chunked critics
• multimodal policy
• sample-efficient fine-tuning
• dexterous manipulation