Neural Attention: A Novel Mechanism for Enhanced Expressive Power in Transformer Models

2025-02-24

AI Summary

Standard Transformer self-attention relies on dot-product similarity, which lacks expressivity for capturing complex, nonlinear token interactions. To address this, we propose Neural Attention—a novel attention mechanism that replaces the fixed dot-product with a learnable feed-forward neural network to compute attention weights. This is the first approach to introduce parameterized nonlinear mapping into attention weight computation while preserving dimensional compatibility and mathematical differentiability, thereby substantially enhancing representational capacity. The mechanism is inherently cross-modal: it achieves consistent gains across NLP and CV benchmarks—reducing perplexity by over 5% on WikiText-103 and significantly improving classification accuracy on CIFAR-10/100. Through rigorous computational complexity analysis and architectural optimization, we ensure both enhanced modeling capability and practical deployability.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Transformer models typically calculate attention matrices using dot products, which have limitations when capturing nonlinear relationships between embedding vectors. We propose Neural Attention, a technique that replaces dot products with feed-forward networks, enabling a more expressive representation of relationships between tokens. This approach modifies only the attention matrix calculation while preserving the matrix dimensions, making it easily adaptable to existing transformer-based architectures. We provide a detailed mathematical justification for why Neural Attention increases representational capacity and conduct controlled experiments to validate this claim. When comparing Neural Attention and Dot-Product Attention, NLP experiments on WikiText-103 show a reduction in perplexity of over 5 percent. Similarly, experiments on CIFAR-10 and CIFAR-100 show comparable improvements for image classification tasks. While Neural Attention introduces higher computational demands, we develop techniques to mitigate these challenges, ensuring practical usability without sacrificing the increased expressivity it provides. This work establishes Neural Attention as an effective means of enhancing the predictive capabilities of transformer models across a variety of applications.

Problem

• Enhancing expressive power in Transformer models
• Replacing dot products with feed-forward networks
• Improving NLP and image classification performance

Innovation

• Replaces dot products with feed-forward networks
• Preserves matrix dimensions in attention calculation
• Enhances transformer model expressivity across applications