AngelSlim: A more accessible, comprehensive, and efficient toolkit for large model compression

2026-02-07

AI Summary

This work addresses the lack of a unified and efficient toolchain for large model compression and industrial deployment, particularly in ultra-low-bit quantization, long-context inference acceleration, and multimodal compression. To this end, we propose AngelSlim, a comprehensive toolkit integrating several innovations: FP8/INT8 post-training quantization, 2-bit ultra-low-bit quantization, training-aligned speculative decoding, hybrid static-dynamic sparse attention, IDPruner for vision token pruning, and Samp for adaptive audio token merging. Using this framework, we achieve the first industrial-grade 2-bit large language model, HY-1.8B-int2, which significantly reduces first-token latency, improves inference throughput by 1.8–2.0×, and maintains output correctness—thereby advancing the practical deployment of highly compressed large models.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

This technical report introduces AngelSlim, a comprehensive and versatile toolkit for large model compression developed by the Tencent Hunyuan team. By consolidating cutting-edge algorithms, including quantization, speculative decoding, token pruning, and distillation. AngelSlim provides a unified pipeline that streamlines the transition from model compression to industrial-scale deployment. To facilitate efficient acceleration, we integrate state-of-the-art FP8 and INT8 Post-Training Quantization (PTQ) algorithms alongside pioneering research in ultra-low-bit regimes, featuring HY-1.8B-int2 as the first industrially viable 2-bit large model. Beyond quantization, we propose a training-aligned speculative decoding framework compatible with multimodal architectures and modern inference engines, achieving 1.8x to 2.0x throughput gains without compromising output correctness. Furthermore, we develop a training-free sparse attention framework that reduces Time-to-First-Token (TTFT) in long-context scenarios by decoupling sparse kernels from model architectures through a hybrid of static patterns and dynamic token selection. For multimodal models, AngelSlim incorporates specialized pruning strategies, namely IDPruner for optimizing vision tokens via Maximal Marginal Relevance and Samp for adaptive audio token merging and pruning. By integrating these compression strategies from low-level implementations, AngelSlim enables algorithm-focused research and tool-assisted deployment.

Problem

• large model compression
• industrial deployment
• efficient inference
• multimodal models
• resource-constrained acceleration

Innovation

• model compression
• post-training quantization
• speculative decoding
• sparse attention
• multimodal pruning