Efficiently Serving LLM Reasoning Programs with Certaindex

2024-12-30

AI Summary

To address the high computational overhead, latency, and service cost induced by complex inference algorithms (e.g., chain-of-thought) in large language models (LLMs), this paper proposes Dynasor, a dynamic inference serving system. Methodologically, Dynasor introduces Certaindex—a lightweight, model-determinism-based confidence metric—to enable fine-grained quantification of inference progress. Leveraging Certaindex, it designs query difficulty-aware dynamic compute allocation and SLO-driven early-exit mechanisms, jointly optimizing accuracy, latency, and cost at the serving layer. The system integrates a dynamic scheduler, inference path tracing, and adaptive batching. Evaluation results demonstrate that Dynasor reduces computation by 50%, improves online throughput by 3.3×, or tightens tail-latency SLOs by 4.7×—significantly enhancing both resource efficiency and service quality.

Technology Category

• Machine Learning: Hardware-aware ML
• Planning, Routing, and Scheduling: Planning with Language Models
• Search and Optimization: Learning to Search

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

The rapid evolution of large language models (LLMs) has unlocked their capabilities in advanced reasoning tasks like mathematical problem-solving, code generation, and legal analysis. Central to this progress are inference-time reasoning algorithms, which refine outputs by exploring multiple solution paths, at the cost of increasing compute demands and response latencies. Existing serving systems fail to adapt to the scaling behaviors of these algorithms or the varying difficulty of queries, leading to inefficient resource use and unmet latency targets. We present Dynasor, a system that optimizes inference-time compute for LLM reasoning queries. Unlike traditional engines, Dynasor tracks and schedules requests within reasoning queries and uses Certaindex, a proxy that measures statistical reasoning progress based on model certainty, to guide compute allocation dynamically. Dynasor co-adapts scheduling with reasoning progress: it allocates more compute to hard queries, reduces compute for simpler ones, and terminates unpromising queries early, balancing accuracy, latency, and cost. On diverse datasets and algorithms, Dynasor reduces compute by up to 50% in batch processing and sustaining 3.3x higher query rates or 4.7x tighter latency SLOs in online serving.

Problem

• Large Language Models
• Resource Efficiency
• Processing Speed

Innovation

• Dynasor
• Dynamic Resource Allocation
• Certaindex Evaluation Model