FineVLA: Fine-Grained Instruction Alignment for Steerable Vision-Language-Action Policies

2026-05-26

AI Summary

Existing robotic datasets provide only coarse-grained task instructions, lacking critical manipulation details—such as the executing arm, approach direction, and contact region—which hinders the learning of controllable vision-language-action (VLA) policies. This work proposes FineVLA, a framework that systematically introduces fine-grained language supervision for the first time, leveraging a human-verified dataset of fine-grained instruction-action alignments and a controllable hybrid training strategy that fuses coarse- and fine-grained instructions. Key contributions include a scalable fine-grained annotation pipeline, a robot-specific vision-language model (VLM) annotator, and a new visual question answering (VQA) evaluation benchmark. Experiments demonstrate that FineVLA achieves success rates of 86.8%/82.5% on RoboTwin simulation and 62.7/100 on real-world bimanual tasks, significantly outperforming baselines, with 18–23 point improvements in controllability across dimensions such as pose, color, and approach direction.

Abstract

Vision-Language-Action (VLA) models are increasingly expected to not only complete robot tasks, but also follow human instructions about how those tasks should be executed. However, existing robot datasets usually pair trajectories with coarse goal-level language, leaving execution-critical details such as active arm, approach direction, and contact region unspecified. This limits steerable policy learning and robotic video understanding. We introduce FineVLA, an open framework for action-aligned fine-grained VLA supervision. The framework includes: (1) a data construction tool that unifies 972,247 trajectories across 85K tasks from 10 open-source robot datasets and builds FineVLA-Data, a human-verified dataset of 47,159 fine-grained trajectories; (2) a held-out benchmark with 500 videos, 10,816 atomic facts, and 1,030 VQA questions; (3) a robotics-specialized VLM annotator for scalable fine-grained annotation; and (4) a steerable VLA policy trained with controlled mixtures of fine-grained and raw goal-level instructions. Our experiments yield three findings. First, fine-grained supervision does not sacrifice goal-level success: FG-only improves over Raw-only by +1.4 to +8.1 success-rate points across settings. Second, fine-grained and raw instructions are complementary, following a consistent inverted-U trend peaking at FG:Raw = 1:2 to 1:1. The best mixed setting reaches 86.8%/82.5% in RoboTwin simulation and 62.7/100 in real-world dual-arm manipulation (vs. 49.9 Raw-only). Third, fine-grained supervision improves steerable control: the largest real-world gains appear on pose (+23), color (+18), and approach direction (+18)--factors where goal-level instructions provide no guidance. Overall, fine-grained language should augment goal-level instructions: specifying how to execute alongside what to achieve. Project page: https://finevla.xlang.ai/

Problem

• Vision-Language-Action
• fine-grained instruction
• robotic policy
• steerable control
• instruction alignment

Innovation

• fine-grained instruction
• vision-language-action
• steerable policy
• robotic dataset
• instruction alignment