LooC: Effective Low-Dimensional Codebook for Compositional Vector Quantization

2026-01-01

arXiv.org

AI Summary

This work addresses the longstanding challenge in vector quantization of simultaneously achieving high capacity and compactness. To this end, the authors propose LooC, a novel approach that employs a low-dimensional compositional codebook wherein code vectors serve as reconstructible units. A parameter-free interpolation-extrapolation mechanism is introduced to enhance feature fidelity without increasing model parameters. This design effectively mitigates codebook collapse, ensures full utilization of the entire codebook, and allows seamless plug-and-play integration into existing vector quantization frameworks. Extensive experiments demonstrate that LooC consistently outperforms current state-of-the-art methods across diverse tasks, datasets, and architectures, achieving superior performance with a significantly smaller codebook size.

Technology Category

• Computer Vision: Learning & Optimization for CV
• Machine Learning: Deep Generative Models & Autoencoders
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Vector quantization (VQ) is a prevalent and fundamental technique that discretizes continuous feature vectors by approximating them using a codebook. As the diversity and complexity of data and models continue to increase, there is an urgent need for high-capacity, yet more compact VQ methods. This paper aims to reconcile this conflict by presenting a new approach called LooC, which utilizes an effective Low-dimensional codebook for Compositional vector quantization. Firstly, LooC introduces a parameter-efficient codebook by reframing the relationship between codevectors and feature vectors, significantly expanding its solution space. Instead of individually matching codevectors with feature vectors, LooC treats them as lower-dimensional compositional units within feature vectors and combines them, resulting in a more compact codebook with improved performance. Secondly, LooC incorporates a parameter-free extrapolation-by-interpolation mechanism to enhance and smooth features during the VQ process, which allows for better preservation of details and fidelity in feature approximation. The design of LooC leads to full codebook usage, effectively utilizing the compact codebook while avoiding the problem of collapse. Thirdly, LooC can serve as a plug-and-play module for existing methods for different downstream tasks based on VQ. Finally, extensive evaluations on different tasks, datasets, and architectures demonstrate that LooC outperforms existing VQ methods, achieving state-of-the-art performance with a significantly smaller codebook.

Problem

• Vector Quantization
• Codebook Compactness
• High-Capacity Representation
• Compositional Quantization
• Feature Approximation

Innovation

• Compositional Vector Quantization
• Low-dimensional Codebook
• Parameter-efficient VQ
• Interpolation-based Extrapolation
• Codebook Collapse Avoidance