HANDFUL: Sequential Grasp-Conditioned Dexterous Manipulation with Resource Awareness

2026-04-27

AI Summary

This work addresses the challenge of task failure in multi-stage dexterous manipulation caused by initial grasps that neglect the requirements of subsequent actions. To this end, the authors propose a resource-aware sequential dexterous manipulation framework that treats fingers as limited resources, deliberately reserving a subset during the initial grasp to support downstream subtasks. The approach integrates a finger-level contact reward mechanism and a curriculum learning strategy to optimize overall performance. As the first study to incorporate resource-aware grasping into sequential manipulation, it introduces HANDFUL-Bench, a novel simulation benchmark, and leverages reinforcement learning combined with sim-to-real transfer techniques. The method demonstrates significantly improved success rates and robustness across diverse subtasks—including pushing, pulling, and pressing—and is validated on the real-world LEAP dexterous hand.

Abstract

Dexterous robot hands offer rich opportunities for multifunctional manipulation, where a robot must execute multiple skills in sequence while maintaining control over previously grasped objects. Most prior work in dexterous manipulation focuses on single-object, single-skill tasks. In contrast, our insight is that many sequential tasks require resource-aware grasps that conserve fingers for future actions. In this paper, we study sequential grasp-conditioned dexterous manipulation, where a robot first grasps an object and then performs a second, distinct manipulation subtask while preserving the initial grasp. We introduce HANDFUL, a learning framework that models finger usage as a limited resource and encourages exploration of resource-aware grasps through finger-level contact rewards. These grasps are subsequently selected for downstream tasks via curriculum-based policy learning. We further propose HANDFUL-Bench, a simulation benchmark that introduces sequential dexterous manipulation tasks across multiple secondsubtask objectives, including pushing, pulling, and pressing, under a shared grasp-conditioned setup. Extensive simulation results demonstrate that prioritizing resource-aware grasps improves second-subtask success and robustness compared to a baseline that greedily optimizes the initial grasp before attempting the second subtask. We additionally validate our approach on a real dexterous LEAP hand. Together, this work establishes resource-aware grasp planning as a key principle for multifunctional dexterous manipulation. Supplementary material is available on our website: https://handful-dex.github.io.

Problem

• dexterous manipulation
• sequential tasks
• grasp-conditioned
• resource awareness
• multifunctional manipulation

Innovation

• resource-aware grasping
• dexterous manipulation
• sequential tasks
• finger-level contact rewards
• curriculum-based policy learning