Large Language Models Meet Extreme Multi-label Classification: Scaling and Multi-modal Framework

2025-11-17

AI Summary

To address the high computational cost of large language models (LLMs) and the limited representational capacity of unimodal text in extreme multi-label classification (XMC), this paper proposes ViXML—the first multimodal XMC framework that efficiently integrates a decoder-only LLM with visual modality. Methodologically, ViXML innovatively combines single-image embedding pooling, vision-augmented data construction, and maximum inner-product search to enable lightweight visual information injection and fast retrieval. Evaluated on four standard XMC benchmarks, ViXML significantly outperforms state-of-the-art methods despite using fewer parameters, achieving an 8.21% improvement in P@1. Notably, it demonstrates superior generalization under low-resource settings. To foster reproducibility and further research, we publicly release both the source code and the first vision-extended dataset specifically designed for XMC.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Data Mining & Knowledge Management: Mining of Visual, Multimedia & Multimodal Data

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Foundation models have revolutionized artificial intelligence across numerous domains, yet their transformative potential remains largely untapped in Extreme Multi-label Classification (XMC). Queries in XMC are associated with relevant labels from extremely large label spaces, where it is critical to strike a balance between efficiency and performance. Therefore, many recent approaches efficiently pose XMC as a maximum inner product search between embeddings learned from small encoder-only transformer architectures. In this paper, we address two important aspects in XMC: how to effectively harness larger decoder-only models, and how to exploit visual information while maintaining computational efficiency. We demonstrate that both play a critical role in XMC separately and can be combined for improved performance. We show that a few billion-size decoder can deliver substantial improvements while keeping computational overhead manageable. Furthermore, our Vision-enhanced eXtreme Multi-label Learning framework (ViXML) efficiently integrates foundation vision models by pooling a single embedding per image. This limits computational growth while unlocking multi-modal capabilities. Remarkably, ViXML with small encoders outperforms text-only decoder in most cases, showing that an image is worth billions of parameters. Finally, we present an extension of existing text-only datasets to exploit visual metadata and make them available for future benchmarking. Comprehensive experiments across four public text-only datasets and their corresponding image enhanced versions validate our proposals' effectiveness, surpassing previous state-of-the-art by up to +8.21% in P@1 on the largest dataset. ViXML's code is available at https://github.com/DiegoOrtego/vixml.

Problem

• Harnessing decoder-only models for Extreme Multi-label Classification
• Integrating visual information while maintaining computational efficiency
• Combining text and vision models to improve classification performance

Innovation

• Utilizing billion-size decoder-only models for XMC
• Integrating vision models via single image embedding pooling
• Combining text and visual data for multi-modal XMC