Bitnet.cpp: Efficient Edge Inference for Ternary LLMs

2025-02-17

AI Summary

This work addresses the lack of efficient inference support for ternary large language models (e.g., BitNet b1.58) on edge devices. We present the first lossless sub-2-bit-per-weight real-time inference system tailored for edge deployment. Our method introduces: (1) a novel hybrid-precision GEMM scheme combining ternary lookup tables (TL) and scaled Int2 (I2_S), balancing numerical fidelity and memory efficiency; (2) a general-purpose, low-bit element-wise lookup table (ELUT) technique enabling arbitrary low-bit nonlinear mappings; and (3) hardware-aware C++ optimizations targeting edge accelerators. Experiments show up to 6.25× speedup over full-precision baselines and up to 2.32× improvement over state-of-the-art low-bit approaches—while incurring zero accuracy degradation. The implementation is open-sourced and supports plug-and-play deployment.

Technology Category

• Machine Learning: Hardware-aware ML
• Search and Optimization: Learning to Search
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

The advent of 1-bit large language models (LLMs), led by BitNet b1.58, has spurred interest in ternary LLMs. Despite this, research and practical applications focusing on efficient edge inference for ternary LLMs remain scarce. To bridge this gap, we introduce Bitnet.cpp, an inference system optimized for BitNet b1.58 and ternary LLMs. Given that mixed-precision matrix multiplication (mpGEMM) constitutes the bulk of inference time in ternary LLMs, Bitnet.cpp incorporates a novel mpGEMM library to facilitate sub-2-bits-per-weight, efficient and lossless inference. The library features two core solutions: Ternary Lookup Table (TL), which addresses spatial inefficiencies of previous bit-wise methods, and Int2 with a Scale (I2_S), which ensures lossless edge inference, both enabling high-speed inference. Our experiments show that Bitnet.cpp achieves up to a 6.25x increase in speed over full-precision baselines and up to 2.32x over low-bit baselines, setting new benchmarks in the field. Additionally, we expand TL to element-wise lookup table (ELUT) for low-bit LLMs in the appendix, presenting both theoretical and empirical evidence of its considerable potential. Bitnet.cpp is publicly available at https://github.com/microsoft/BitNet/tree/paper , offering a sophisticated solution for the efficient and practical deployment of edge LLMs.

Problem

• Efficient edge inference for ternary LLMs
• Optimized mpGEMM for sub-2-bits-per-weight inference
• High-speed, lossless edge inference solutions

Innovation

• Optimized inference for ternary LLMs
• Novel mpGEMM library for efficiency
• Ternary Lookup Table for spatial efficiency