The Geometry of Robustness: Optimizing Loss Landscape Curvature and Feature Manifold Alignment for Robust Finetuning of Vision-Language Models

2026-03-28

AI Summary

This work addresses the challenge of simultaneously achieving high in-distribution accuracy, out-of-distribution (OOD) generalization, and adversarial robustness in visual-language model fine-tuning. To this end, the authors propose GRACE, a novel framework that unifies the modeling of flatness in parameter space and invariance in feature space—a first in the field. GRACE integrates an adaptive curvature-aware weight perturbation mechanism with a feature Gram matrix alignment loss across clean, adversarial, and OOD inputs. Grounded in Robust PAC-Bayes theory, this approach enables joint optimization of all three performance aspects. Experiments on CLIP fine-tuned on ImageNet demonstrate significant gains: a 10.8% improvement in in-distribution accuracy, a 13.5% increase in adversarial accuracy, and a maintained OOD accuracy of 57.0%.

Abstract

Fine-tuning approaches for Vision-Language Models (VLMs) face a critical three-way trade-off between In-Distribution (ID) accuracy, Out-of-Distribution (OOD) generalization, and adversarial robustness. Existing robust fine-tuning strategies resolve at most two axes of this trade-off. Generalization-preserving methods retain ID/OOD performance but leave models vulnerable to adversarial attacks, while adversarial training improves robustness to targeted attacks but degrades ID/OOD accuracy. Our key insight is that the robustness trade-off stems from two geometric failures: sharp, anisotropic minima in parameter space and unstable feature representations that deform under perturbation. To address this, we propose GRACE (Gram-aligned Robustness via Adaptive Curvature Estimation), a unified fine-tuning framework that jointly regularizes the parameter-space curvature and feature-space invariance for VLMs. Grounded in Robust PAC-Bayes theory, GRACE employs adaptive weight perturbations scaled by local curvature to promote flatter minima, combined with a feature alignment loss that maintains representation consistency across clean, adversarial, and OOD inputs. On ImageNet fine-tuning of CLIP models, GRACE simultaneously improves ID accuracy by 10.8%, and adversarial accuracy by 13.5% while maintaining 57.0% OOD accuracy (vs. 57.4% zero-shot baseline). Geometric analysis confirms that GRACE converges to flatter minima without feature distortion across distribution shifts, providing a principled step toward generalized robustness in foundation VLMs.

Problem

• robust fine-tuning
• vision-language models
• distribution shift
• adversarial robustness
• generalization

Innovation

• loss landscape curvature
• feature manifold alignment
• robust fine-tuning
• adaptive curvature estimation
• vision-language models