🤖 AI Summary
This work addresses two key challenges in large language model (LLM) research: the difficulty of training data provenance attribution and low efficiency in lossless text compression. We propose Llamazip—a unified, LLaMA3-based framework for joint lossless text compression and training set membership inference. Leveraging LLaMA3’s token-level predictive capability, Llamazip encodes only residual tokens that the model fails to predict accurately, achieving high compression efficiency. Crucially, the model’s per-token prediction confidence distribution serves as a discriminative signal for membership inference—marking the first direct use of LLM memorization for training set membership detection. Through quantized compression, context window optimization, and systematic evaluation, Llamazip achieves superior compression ratios over conventional algorithms on standard text corpora and attains high membership detection accuracy (AUC > 0.92). This work establishes a novel paradigm for enhancing model transparency, safeguarding data copyright, and enabling perception-aware LLM compression.
📝 Abstract
This work introduces Llamazip, a novel lossless text compression algorithm based on the predictive capabilities of the LLaMA3 language model. Llamazip achieves significant data reduction by only storing tokens that the model fails to predict, optimizing storage efficiency without compromising data integrity. Key factors affecting its performance, including quantization and context window size, are analyzed, revealing their impact on compression ratios and computational requirements. Beyond compression, Llamazip demonstrates the potential to identify whether a document was part of the training dataset of a language model. This capability addresses critical concerns about data provenance, intellectual property, and transparency in language model training.