ScAle: Attention Head Scaling as a Minimal Adapter for Spatial Reasoning in Vision Language Models

2026-06-28

AI Summary

This work addresses the limited performance of vision-language models on spatial reasoning tasks by proposing an ultra-lightweight adaptation method. Without modifying or fine-tuning any pre-trained weights, the approach freezes the entire backbone and learns only approximately 1,000 scalar coefficients to reweight the outputs of attention heads and MLP activations corresponding to the final token. This strategy achieves both architectural generality and parameter efficiency, yielding substantial improvements across multiple benchmarks—including SpatialEval, COCOQA, and VGQA—with relative accuracy gains of up to 134.1%. Remarkably, it recovers a large portion of the performance typically attained by standard parameter-efficient fine-tuning (PEFT) methods while incurring minimal parameter overhead.

Abstract

Spatial reasoning remains a persistent challenge for many vision language models (VLMs), and improving it typically requires fine-tuning with substantial additional parameters. Our preliminary analysis reveals that rescaling activations in selected transformer layers-without modifying pretrained weights-can significantly influence downstream performance. Motivated by this observation, we propose ScAle, an ultra-lightweight adaptation method that learns a small set of scalar coefficients to modulate last-token attention and MLP activations in a fully frozen backbone. We evaluate our method on the synthetic spatial reasoning benchmark SpatialEval and on real-world VQA datasets (COCOQA and VGQA) across multiple model families. Our method, ScAle, achieves up to 134.1% relative accuracy gains using only 1K trainable parameters without requiring millions of trainable parameters as in standard PEFT methods such as LoRA. Despite its extreme compactness, our approach recovers a substantial fraction of standard PEFT performance while preserving strong non-spatial VQA accuracy. These results demonstrate that bounded activation reweighting provides a simple, architecture-agnostic, and highly parameter-efficient alternative for adapting pretrained VLMs.

Problem

• spatial reasoning
• vision language models
• parameter-efficient adaptation
• activation scaling
• frozen backbone

Innovation

• activation scaling
• parameter-efficient adaptation
• spatial reasoning
• vision-language models
• frozen backbone