Vision-LSTM: xLSTM as Generic Vision Backbone

2024-06-06

arXiv.org

AI Summary

Vision backbones overly rely on Transformers, while conventional LSTMs suffer from poor parallelizability and weak long-range modeling capability. Method: We propose Vision-LSTM (ViL), the first general-purpose vision backbone based on scalable xLSTM—featuring exponential gating and parallelized matrix memory—adapted for image patch sequence modeling. ViL introduces a novel bidirectional alternating processing mechanism to jointly capture local details and global dependencies, combined with lightweight patch embedding and sequential modeling. Contribution/Results: ViL achieves significantly enhanced representational capacity with low computational overhead. Experiments demonstrate that ViL consistently outperforms LSTM-based baselines on ImageNet classification, COCO object detection, and ADE20K semantic segmentation, while approaching the performance of state-of-the-art Vision Transformers (ViTs). This validates ViL as a highly efficient, general-purpose visual backbone with strong practical potential.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Transformers are widely used as generic backbones in computer vision, despite initially introduced for natural language processing. Recently, the Long Short-Term Memory (LSTM) has been extended to a scalable and performant architecture - the xLSTM - which overcomes long-standing LSTM limitations via exponential gating and parallelizable matrix memory structure. In this report, we introduce Vision-LSTM (ViL), an adaption of the xLSTM building blocks to computer vision. ViL comprises a stack of xLSTM blocks where odd blocks process the sequence of patch tokens from top to bottom while even blocks go from bottom to top. Experiments show that ViL holds promise to be further deployed as new generic backbone for computer vision architectures.

Problem

• Adapts xLSTM for computer vision tasks
• Overcomes LSTM limitations via new architecture
• Proposes Vision-LSTM as a new vision backbone

Innovation

• xLSTM adaptation to vision
• Exponential gating enhances performance
• Alternating sequence processing blocks