BitNet b1.58 2B4T Technical Report

2025-04-16

AI Summary

To address the high computational cost of large language models (LLMs), this work introduces the first open-source, natively 1-bit (b1.58-weight-quantized) 2-billion-parameter LLM, trained end-to-end on 4 trillion tokens. We achieve stable training and inference with purely 1-bit weights and 1.58-bit activations at the 2B scale—enabled by a custom gradient propagation mechanism and hardware-aware sparse tensor operations. We further release a cross-platform (GPU/CPU) open-source inference engine. Compared to full-precision baselines, our model reduces memory footprint by 8×, cuts decoding latency by 60%, and significantly lowers energy consumption. It matches or exceeds the performance of Llama-2-2B and Phi-3-2B across language understanding, mathematical reasoning, code generation, and dialogue tasks—demonstrating a breakthrough in the accuracy–efficiency trade-off for ultra-low-bit LLMs.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

We introduce BitNet b1.58 2B4T, the first open-source, native 1-bit Large Language Model (LLM) at the 2-billion parameter scale. Trained on a corpus of 4 trillion tokens, the model has been rigorously evaluated across benchmarks covering language understanding, mathematical reasoning, coding proficiency, and conversational ability. Our results demonstrate that BitNet b1.58 2B4T achieves performance on par with leading open-weight, full-precision LLMs of similar size, while offering significant advantages in computational efficiency, including substantially reduced memory footprint, energy consumption, and decoding latency. To facilitate further research and adoption, the model weights are released via Hugging Face along with open-source inference implementations for both GPU and CPU architectures.

Problem

• Develop first open-source 1-bit 2B parameter LLM
• Evaluate model on language, math, coding, conversation tasks
• Achieve efficiency gains vs full-precision similar-size LLMs

Innovation

• First 1-bit LLM at 2B scale
• Trained on 4 trillion tokens
• Efficient with low resource usage