Scaling LLM Knowledge Boundaries via Distribution-Optimized Synthesis

2026-06-22

AI Summary

This work addresses the limitation of existing synthetic data generation methods, which often neglect knowledge distribution and consequently lead to imbalanced knowledge injection that constrains the expansion of large language models’ knowledge boundaries. To overcome this, the authors propose KDoS, a novel framework that introduces, for the first time, a knowledge distribution optimization hypothesis. KDoS leverages a knowledge density metric and a three-stage feedback mechanism to enable distribution-aware synthetic data generation. Extensive experiments demonstrate that KDoS consistently outperforms baseline methods across six knowledge benchmarks, with model sizes ranging from 0.6B to 16B parameters and training data scales from 1B to 5B tokens. These results validate that an optimized knowledge distribution can stably enhance a model’s knowledge boundary and exhibits strong generalization across both model architectures and data scales.

Abstract

Knowledge injection via synthetic data is crucial for enhancing Large Language Models (LLMs). However, current synthesis methods simply stop at preset token counts or fixed data ratios, lacking awareness of knowledge distribution. This results in some domains being sparse while others are redundant, limiting LLM knowledge boundaries. We revisit knowledge injection from a distribution perspective and hypothesize that an optimal knowledge distribution exists to maximize knowledge boundary expansion. We propose KDoS (Knowledge Distribution-optimized Synthesis), a framework that introduces knowledge density to drive synthesis through a three-stage feedback mechanism, shifting from blind generation to distribution-optimized synthesis. We construct Wikipedia-based synthetic data with varying knowledge distributions and conduct experiments on models from 0.6B to 16B (Qwen, Ling, LLaMA) and data scales from 1B to 5B tokens. Our key findings are: (1) an optimal knowledge distribution consistently maximizes boundary expansion; (2) this distribution is stable across backbones and scales; (3) KDoS outperforms baselines across six knowledge benchmarks. Our work offers a new perspective and practical framework for synthetic data-driven knowledge injection.

Problem

• synthetic data
• knowledge injection
• knowledge distribution
• Large Language Models
• knowledge boundaries

Innovation

• knowledge injection
• synthetic data
• knowledge distribution
• distribution optimization
• large language models