From Production Traffic to Post-Training: Building a Self-Hosted LLM That Covers the Corporate Request Mix

2026-09-01

AI Summary

为解决企业自托管LLM导致的GPU资源碎片化问题，通过分析生产错误并训练特定领域的专家模型，使用两阶段SLERP方法合并，提高了模型性能和效率。

Abstract

Data-residency constraints force enterprises to self-host LLMs, but continuous adoption of newer models without decommissioning their predecessors expands the serving fleet, fragmenting a finite GPU pool. We consolidate traffic from over 200 internal applications onto a single model by closing quality gaps identified through production error analysis along three axes: instruction following, function-calling, and internal task distribution. Quality is tracked by offline benchmarks stratified to production traffic and scored by deterministic verifiers or calibrated LLM judges. Rather than optimising all objectives jointly, which introduces cross-domain reward interference, we train a separate GRPO expert per axis and merge them via two-stage SLERP. Each expert's reward exposes a distinct failure mode, namely semantic collapse, over-calling, and verbosity hacking, each requiring a domain-specific fix. In non-reasoning mode the recipe surpasses a ${\sim}7\times$ larger by total parameters baseline on the in-house Arena with 69.6 to 65.8, instruction following with 0.85 to 0.83, and function-calling with 0.79 to 0.77, while lifting general dialogue benchmarks. The model absorbs 50% of platform traffic, 116M requests per month, at a fraction of the serving cost.

Problem

• data-residency
• self-hosted LLMs
• GPU fragmentation
• production traffic

Innovation

• production error analysis
• GRPO expert
• two-stage SLERP