Output-Level Regularization Eliminates the Seed Lottery in Single-GPU VLA Fine-Tuning

2026-06-11

AI Summary

This study addresses the severe performance instability—termed the “seed lottery” problem—in single-GPU fine-tuning of vision–language–action (VLA) models, where success rates plummet from 91–94% to 65.2% across different random seeds. The authors identify, for the first time, that this collapse stems from unconstrained weight movement within the null space of the output layer’s Jacobian, leading to output degeneracy. To mitigate this, they propose an output-layer regularization strategy combining VICReg, Dropout, and halved learning rates. Evaluated on the VLA-JEPA model across three LIBERO benchmarks, their method completely eliminates collapse in all 21 independent runs (0/21), significantly outperforming the baseline (1/13), with F(12,11) = 28.7, p < 0.001. Notably, conventional weight-level regularization proves ineffective.

Abstract

Fine-tuning a vision-language-action model (VLA-JEPA) on a single GPU should be simple: load a pretrained checkpoint, run training, deploy. There is a hidden danger. Run the same fine-tuning code thirteen times -- same data, same architecture, different random seed -- and twelve runs produce a robot succeeding 91--94% of the time, while one run silently degrades to 65.2%: a 29 pp gap with no error message, no warning, and no way to predict which seed will fail. We call this the seed lottery. We trace the cause to output collapse: the action predictor quietly learns to produce nearly identical outputs regardless of what the robot sees. Existing weight-level methods (L2, EWC) are structurally blind to this collapse -- they penalize weight changes, but collapse occurs in directions weights can move freely without affecting outputs, a gap we formalize via the Jacobian null-space. Across 7 methods x up to 13 seeds x 3 LIBERO benchmarks, three output-level regularizers -- VICReg (n=12 seeds), Dropout (n=4), and a halved learning rate (n=5) -- each eliminate every catastrophic seed (0/21 combined collapses vs. 1/13 Baseline; F(12,11)=28.7, p<0.001), while weight-level methods (L2, EWC) preserve the lottery. The simplest fix is changing one number in your optimizer config.

Problem

• seed lottery
• output collapse
• vision-language-action model
• fine-tuning instability
• random seed sensitivity

Innovation

• output-level regularization
• seed lottery
• output collapse
• VLA fine-tuning
• Jacobian null-space