Isotropic Embedding Perturbations for Robust Vision Language Encoders

2026-09-09

AI Summary

本文提出Aether方法，通过在嵌入空间而非输入空间中应用随机扰动来解决多模态对齐问题，提高视觉-语言编码器的鲁棒性。

Abstract

Data augmentation is fundamental to training modern deep vision and multimodal models. While individual methods, such as RandAug, CutMix, Mixup, RandErase, and DropPath, offer strong regularization effects, their combined use has saturated in performance due to overlapping functionalities, and aggressive pixel-level manipulations may disrupt delicate cross-modal alignment. This saturation motivates the search for a new augmentation axis within the embedding space rather than the input space. We introduce Aether, a simple plug-in method that applies diffusion-style random perturbations in the embedding space via controlled alpha-mixing, specifically designed to provide isotropic regularization that remains semantically consistent. Inspired by feature-space perturbations in language models and image degradation in generative pretraining, Aether induces mild yet effective perturbations that smooth the representations without compromising the fine-grained structural information required for strong vision-language encoders. Across diverse architectures and across multiple recognition tasks, Aether delivers consistent gains over the advanced recipe combining CutMix, Mixup, DropPath, and RandAug---a level of improvement rarely observed with modern augmentation alternatives. Notably, Aether demonstrates superior effectiveness in multi-modal alignment, succeeding where traditional pixel-space augmentations fail by providing a stable, isotropic regularization signal that respects the integrity of the high-dimensional feature space.

Problem

• Data Augmentation
• Vision-Language Encoders
• Embedding Space
• Regularization
• Cross-Modal Alignment

Innovation

• isotropic regularization
• embedding space perturbation
• controlled alpha-mixing
• multi-modal alignment