Contrastive Weak-to-strong Generalization

2025-10-09

AI Summary

Weak models’ outputs suffer from severe noise and bias, critically undermining the robustness and practicality of weak-to-strong generalization. To address this, we propose ConG—a novel framework that establishes, for the first time, the structural equivalence between implicit reward modeling and contrastive decoding, thereby enabling a new weak-to-strong generalization paradigm without human feedback or explicit reward models. ConG aligns weak-model generations via contrastive decoding and jointly performs implicit reward modeling and log-likelihood ratio estimation to suppress output noise and enable robust capability transfer. Extensive experiments across major LLM families—including LLaMA, Qwen, and Phi—demonstrate that ConG consistently and significantly improves generalization stability and downstream task performance, outperforming state-of-the-art methods across all benchmarks. These results validate ConG’s architectural generality and empirical effectiveness.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Weak-to-strong generalization provides a promising paradigm for scaling large language models (LLMs) by training stronger models on samples from aligned weaker ones, without requiring human feedback or explicit reward modeling. However, its robustness and generalization are hindered by the noise and biases in weak-model outputs, which limit its applicability in practice. To address this challenge, we leverage implicit rewards, which approximate explicit rewards through log-likelihood ratios, and reveal their structural equivalence with Contrastive Decoding (CD), a decoding strategy shown to reduce noise in LLM generation. Building on this connection, we propose Contrastive Weak-to-Strong Generalization (ConG), a framework that employs contrastive decoding between pre- and post-alignment weak models to generate higher-quality samples. This approach enables more reliable capability transfer, denoising, and improved robustness, substantially mitigating the limitations of traditional weak-to-strong methods. Empirical results across different model families confirm consistent improvements, demonstrating the generality and effectiveness of ConG. Taken together, our findings highlight the potential of ConG to advance weak-to-strong generalization and provide a promising pathway toward AGI.

Problem

• Addressing noise and bias in weak-to-strong generalization
• Improving robustness and generalization of weak-to-strong methods
• Generating higher-quality samples for reliable capability transfer

Innovation

• Uses contrastive decoding between pre-alignment models
• Generates higher-quality samples from weak models
• Improves robustness and denoising in generalization