TRIM: Token Reduction and Inference Modeling for Cost-Effective Language Generation

2024-12-10

arXiv.org

AI Summary

To address the prohibitively high inference cost of large language models (LLMs) for long-text generation, this paper proposes a two-stage distillation inference paradigm. In the first stage, prompt engineering guides an LLM to produce semantically condensed yet information-complete “distilled outputs”; in the second stage, a lightweight fine-tuned Transformer efficiently reconstructs these outputs into full-length text. This approach is the first to decouple semantic compression from textual reconstruction, thereby disassociating computational overhead from output length. Experiments on general-knowledge domains show an average 20.58% reduction in generated tokens, with only marginal degradation in BLEU and ROUGE scores (<1.2 points), yielding substantial gains in generation throughput per unit compute. The core contribution lies in establishing a scalable “LLM distillation + small-model reconstruction” paradigm, underpinned by principled prompt design, sequence-to-sequence modeling, and evaluation alignment to ensure semantic fidelity.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

The inference cost of Large Language Models (LLMs) is a significant challenge due to their computational demands, specially on tasks requiring long outputs. However, natural language often contains redundancy, which presents an opportunity for optimization. We have observed that LLMs can generate distilled language-concise outputs that retain essential meaning, when prompted appropriately. We propose TRIM, a pipeline for saving computational cost in which a shorter distilled output from the LLM is reconstructed into a full narrative by a smaller model with lower inference costs. Our experiments show promising results, particularly in general knowledge domains with 20.58% saved tokens on average with tiny decrease in evaluation metrics, hinting that this approach can effectively balance efficiency and accuracy in language processing tasks.

Problem

• Reducing computational cost of LLM inference
• Optimizing redundant natural language outputs
• Balancing efficiency and accuracy in generation

Innovation

• Token reduction via distilled concise outputs
• Smaller model reconstructs full narrative
• Balances efficiency and accuracy effectively