Flatness Preserves Instruction Following in Vision-Language-Action Models

2026-06-22

AI Summary

This work addresses the “instruction blindness” problem in vision-language-action (VLA) models, where fine-tuned agents overly rely on visual shortcuts and neglect linguistic instructions. To mitigate this, the study introduces flatness-aware optimization into VLA fine-tuning by employing Sharpness-Aware Minimization (SAM) to learn flatter loss landscapes, thereby enhancing robustness to weight perturbations and significantly improving instruction-following capabilities. The proposed approach requires no additional data, architectural modifications, or retraining, and it complements existing guidance techniques. Evaluated across multiple simulated and real-world benchmarks, the method achieves over a 60% improvement in instruction adherence, demonstrating its effectiveness and practicality for deploying more reliable VLA systems.

Abstract

Vision-language-action (VLA) models have the potential for open-world generalization by leveraging pretrained vision-language representations, yet downstream finetuning on limited robot data often degrades these representations, leading to brittle policies that ignore language instructions in favor of visual shortcuts, a failure mode we term instruction blindness. We hypothesize that standard finetuning with limited data applies gradients to a sparse set of points, which manifests as a sharp loss landscape with high-curvature minima. We propose to address this directly through flatness-preserving optimization while finetuning on the exact same data, where learning a flatter landscape results in a model more robust to perturbations in the weight space. Specifically, we demonstrate that simply applying sharpness-aware minimization during VLA finetuning significantly improves instruction following by over 60% across multiple simulation and real-world benchmarks without additional data, architectural modification, or retraining. We further analyze the effect of selective sharpness, quantify its effects, and show that our approach is complementary to existing guidance techniques. Project page can be found at https://haochenz11.github.io/papers/flatness-vla/.

Problem

• instruction blindness
• vision-language-action models
• downstream finetuning
• representation degradation
• language instruction following

Innovation

• flatness-preserving optimization
• instruction blindness
• sharpness-aware minimization
• vision-language-action models
• loss landscape