Rectified Lagrangian for Out-of-Distribution Detection in Modern Hopfield Networks

2025-02-19

AI Summary

Modern Hopfield Networks (MHNs) lack explicit modeling capabilities for out-of-distribution (OOD) samples, limiting their OOD detection performance. To address this, we propose Regularized Lagrangian (RegLag), the first method to introduce *trivial-point attractors*—guaranteed for any interaction matrix—into MHN dynamics as a theoretically grounded OOD criterion. RegLag couples this with differentiable density estimation to enable end-to-end optimization of in-distribution (ID)/OOD binary discrimination. Crucially, it requires no auxiliary networks or pretraining, ensuring both theoretical rigor and computational efficiency. Evaluated across nine image benchmarks, RegLag consistently outperforms state-of-the-art energy-based OOD detectors—including recent Hopfield-energy approaches—achieving an average improvement of 5.2% in detection accuracy. By unifying dynamical attractor analysis with probabilistic density modeling within the MHN framework, RegLag establishes a novel paradigm for robust MHN-based uncertainty quantification and distributional generalization.

Technology Category

• Machine Learning: Other Foundations of Machine Learning
• Search and Optimization: Learning to Search
• Data Mining & Knowledge Management: Anomaly/Outlier Detection

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Modern Hopfield networks (MHNs) have recently gained significant attention in the field of artificial intelligence because they can store and retrieve a large set of patterns with an exponentially large memory capacity. A MHN is generally a dynamical system defined with Lagrangians of memory and feature neurons, where memories associated with in-distribution (ID) samples are represented by attractors in the feature space. One major problem in existing MHNs lies in managing out-of-distribution (OOD) samples because it was originally assumed that all samples are ID samples. To address this, we propose the rectified Lagrangian (RegLag), a new Lagrangian for memory neurons that explicitly incorporates an attractor for OOD samples in the dynamical system of MHNs. RecLag creates a trivial point attractor for any interaction matrix, enabling OOD detection by identifying samples that fall into this attractor as OOD. The interaction matrix is optimized so that the probability densities can be estimated to identify ID/OOD. We demonstrate the effectiveness of RecLag-based MHNs compared to energy-based OOD detection methods, including those using state-of-the-art Hopfield energies, across nine image datasets.

Problem

• Out-of-distribution detection
• Rectified Lagrangian
• Modern Hopfield Networks

Innovation

• Rectified Lagrangian for OOD
• Trivial point attractor creation
• Optimized interaction matrix densities