TIGER: Inverting Transformer Gradients via Embedding-Subspace Distance Optimization

2026-06-16

AI Summary

In federated learning, gradient updates of Transformer models can inadvertently leak clients’ input data. Existing gradient inversion attacks struggle under practical defenses such as quantization or differential privacy and fail to scale to encoder-decoder architectures. This work proposes a continuous gradient inversion attack that, for the first time, leverages the low-rank subspace structure of attention gradients by formulating it as a differentiable optimization objective. Instead of matching full gradients or performing discrete token search, the method directly optimizes token embeddings to align with the true embedding subspace. This approach substantially improves both reconstruction quality and computational efficiency, significantly outperforming prior attacks on encoder-only models and, notably, achieving the first successful reconstruction of decoder inputs under differential privacy, demonstrating exceptional robustness.

Abstract

Federated learning allows multiple clients to jointly train a shared model by sending gradient updates to a central server while keeping raw inputs local. However, prior gradient inversion attacks show that these updates can reveal enough information to reconstruct client inputs. Existing attacks on transformers either optimize dummy inputs to match the true client updates, which is costly and unstable for modern models, or exploit the low rank of attention gradients to identify a subspace containing the true layer embeddings, followed by a discrete membership test for candidate tokens. However, this token test is brittle under numerical noise, i.e., from quantization or Differential Privacy (DP), and scales poorly for encoder models with non-causal attention. We introduce TIGER, a continuous gradient inversion attack that turns this subspace signal into a differentiable objective. Instead of searching over tokens or matching full gradients, TIGER directly optimizes token embeddings to minimize their distance to the subspace. Our experiments demonstrate that on encoder-only models, TIGER substantially improves both reconstruction quality and runtime over existing attacks, while on decoder models, TIGER is more robust than prior subspace-based attacks, enabling the first successful reconstructions in DP-defended federated learning settings.

Problem

• gradient inversion
• federated learning
• transformer
• differential privacy
• embedding subspace

Innovation

• gradient inversion
• embedding-subspace optimization
• federated learning
• transformer attacks
• differential privacy