Test-Time Steering for Lossless Text Compression via Weighted Product of Experts

2025-11-04

Conference on Empirical Methods in Natural Language Processing

AI Summary

Traditional neural text compressors suffer from poor generalization to unseen data distributions, while generic compressors (e.g., gzip) achieve suboptimal compression ratios. To address this, we propose a test-time steering framework that dynamically fuses pretrained autoregressive language models (ARLMs) with classical compressors via weighted product-of-experts (wPoE), without requiring fine-tuning or architectural modification. The fusion weights are adaptively optimized during inference to maximize compression efficiency per input. This work is the first to introduce steering mechanisms into lossless text compression, guaranteeing compression ratios no worse than those of any constituent baseline—thus ensuring strict performance safety. The method incurs negligible computational overhead, maintains broad compatibility across diverse ARLMs, and exhibits strong cross-distribution robustness. Experiments across multiple text domains demonstrate consistent and significant compression ratio improvements over state-of-the-art baselines, validating its plug-and-play applicability.

Technology Category

• Machine Learning: Learning on the Edge & Model Compression
• Data Mining & Knowledge Management: Data Compression
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Lossless compression techniques are crucial in an era of rapidly growing data. Traditional universal compressors like gzip offer low computational overhead, high speed, and broad applicability across data distributions. However, they often lead to worse compression rates than modern neural compressors, which leverage large-scale training data to model data distributions more effectively. Despite their advantages, neural compressors struggle to generalize to unseen data. To address this limitation, we propose a novel framework that performs Test-Time Steering via a Weighted Product of Experts (wPoE). At inference, our method adaptively combines a universal compression model with a pretrained neural language model, ensuring the compression rate is at least as good as that of the best individual model. Extensive experiments demonstrate that our approach improves the performance of text compression without requiring fine-tuning. Furthermore, it seamlessly integrates with any autoregressive language model, providing a practical solution for enhancing text compression across diverse data distributions.

Problem

• Improving neural compressor generalization to unseen data distributions
• Combining universal and neural compressors for optimal compression rates
• Enhancing text compression without fine-tuning across diverse data

Innovation

• Test-time steering via weighted product of experts
• Adaptively combines universal and neural compression models
• Ensures compression rate matches best individual model