Dense Associative Memory with Epanechnikov Energy

2025-06-12

AI Summary

This work addresses the bottleneck in energy function design for Dense Associative Memory (DenseAM) networks. We propose a novel log-sum-ReLU (LSR) energy function, constructed from the Epanechnikov kernel, which departs from the conventional paradigm relying on exponential separation functions. Our method achieves exponential memory capacity without compromising pattern retrieval accuracy and, for the first time in DenseAM, introduces numerous high-likelihood “emergent” local minima—enabling both perfect memory recall and creative memory generation. Theoretical analysis shows that the non-convex energy landscape exhibits superior separability. Experiments on image data demonstrate that LSR significantly outperforms the log-sum-exp (LSE) baseline in memory capacity and emergent memory novelty, while maintaining comparable log-likelihood performance.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Search and Optimization: Learning to Search
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

We propose a novel energy function for Dense Associative Memory (DenseAM) networks, the log-sum-ReLU (LSR), inspired by optimal kernel density estimation. Unlike the common log-sum-exponential (LSE) function, LSR is based on the Epanechnikov kernel and enables exact memory retrieval with exponential capacity without requiring exponential separation functions. Moreover, it introduces abundant additional emph{emergent} local minima while preserving perfect pattern recovery -- a characteristic previously unseen in DenseAM literature. Empirical results show that LSR energy has significantly more local minima (memories) that have comparable log-likelihood to LSE-based models. Analysis of LSR's emergent memories on image datasets reveals a degree of creativity and novelty, hinting at this method's potential for both large-scale memory storage and generative tasks.

Problem

• Proposes novel energy function for DenseAM networks
• Enables exact memory retrieval with exponential capacity
• Introduces emergent local minima preserving pattern recovery

Innovation

• Epanechnikov kernel-based energy function
• Exact memory retrieval with exponential capacity
• Abundant emergent local minima for storage