FutureFill: Fast Generation from Convolutional Sequence Models

2024-10-02

arXiv.org

AI Summary

To address the high time complexity (O(L²)) and large cache overhead (O(L²)) of autoregressive generation in convolutional sequence models, this paper proposes FutureFill. FutureFill is the first method to achieve near-linear generation complexity (O(L log L)) for convolutional models, enabled by a causality-aware convolution kernel decomposition and an incremental state update mechanism that eliminates redundant computation. Additionally, it introduces a minimal caching strategy that compresses the prefill cache to O(L), scaling linearly with sequence length. Theoretical analysis and synthetic task evaluations demonstrate that FutureFill accelerates generation by 3.2–5.8× and reduces memory footprint by one to two orders of magnitude, significantly outperforming state-of-the-art convolutional and Transformer baselines—while preserving modeling capacity and overcoming the long-standing inference efficiency bottleneck of convolutional architectures.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Diffusion Models for Vision

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

We address the challenge of efficient auto-regressive generation in sequence prediction models by introducing FutureFill - a method for fast generation that applies to any sequence prediction algorithm based on convolutional operators. Our approach reduces the generation time requirement from quadratic to quasilinear relative to the context length. Additionally, FutureFill requires a prefill cache sized only by the number of tokens generated, which is smaller than the cache requirements for standard convolutional and attention-based models. We validate our theoretical findings with experimental evidence demonstrating correctness and efficiency gains in a synthetic generation task.

Problem

• Efficient auto-regressive generation in sequence models
• Reduce generation time from quadratic to quasilinear
• Minimize prefill cache size for prompt-based generation

Innovation

• Fast generation with convolutional sequence models
• Reduces time from quadratic to quasilinear
• Smaller prefill cache than standard models