Balancing Fidelity and Diversity in Diffusion Models via Symmetric Attention Decomposition: Hopfield Perspective

2026-05-26

AI Summary

This study addresses the longstanding challenge in diffusion models of simultaneously achieving high generation fidelity and diversity. It offers a novel interpretation of the Transformer attention mechanism through the lens of Hopfield networks, decomposing the attention matrix into symmetric and antisymmetric components that respectively model the stable structures of the energy landscape and dynamic cyclic behaviors. Building on this decomposition, the work proposes a tunable mechanism that flexibly balances fidelity and diversity during generation by modulating the weight of the symmetric component. The research establishes a significant correlation between a Hopfield-based stability metric and generation quality, and experimental results demonstrate the method’s effective control over the trade-off between fidelity and diversity.

Abstract

We characterize the pre-softmax attention matrix $\mathbf{QK^\top}$ in transformers as an associative memory matrix encoding pairwise associations between input features. By decomposing this matrix into its symmetric and skew-symmetric parts, we interpret the symmetric component as governing the structure of the energy landscape, and the skew-symmetric component as driving circulation on that landscape. Leveraging the energy formulation induced by the symmetric component, we derive Hopfield-style stability measures that quantify the stability of retrieved features. We observe meaningful correlations between Hopfield-style stability measures and the fidelity-diversity trade-offs in generation. Finally, we propose a controllable knob to modulate this trade-off by modifying the circulation of the underlying dynamics. Code is available at our GitHub (https://github.com/hyeon-cho/Attention-Symmetric-Decomposition).

Problem

• diffusion models
• fidelity-diversity trade-off
• attention mechanism
• Hopfield networks
• generative modeling

Innovation

• Symmetric Attention Decomposition
• Hopfield Energy Landscape
• Fidelity-Diversity Trade-off
• Skew-Symmetric Dynamics
• Controllable Generation