Maximizing mutual information between prompts and responses improve LLM personalization with no additional data or human oversight

2026-03-10

Abstract

While post-training has successfully improved large language models (LLMs) across a variety of domains, these gains heavily rely on human-labeled data or external verifiers. Existing data has already been exploited, and new high-quality data is expensive to collect. More fundamentally, true intelligence goes far beyond tasks that are easily verifiable. Therefore, we need self-improvement frameworks that allow models to improve without heavily relying on external oversight. We propose Mutual Information Preference Optimization (MIPO), a contrastive data augmentation method that constructs preference pairs by generating a positive response conditioning on the correct prompt, and a negative response by conditioning on a random, unrelated prompt. We show that using Direct Preference Optimization (DPO) to learn from this paired data maximizes pointwise conditional mutual information (MI), under the base LLM, between prompts and model responses. Empirical results with various-sized Llama- and Qwen-Instruct models show that when used to maximize MI between user context and response, MIPO provides an effective personalization technique, achieving 3-40% gains on personalized instruction-following compared to strong prompting baselines. Surprisingly, MIPO can also be applied to math and multiple-choice problem solving, yielding 1-18% gains without any additional data or human supervision. These results suggest a promising direction for self-improvement using intrinsic signals derived from contrastive data pairs.

Problem

• personalization
• self-improvement
• mutual information
• large language models
• human supervision

Innovation

• Mutual Information
• Preference Optimization
• Self-Improvement
• Contrastive Data Augmentation
• Large Language Models