Self-Improving Model Steering

2025-07-11

AI Summary

Traditional model guidance methods heavily rely on external annotated data, resulting in poor generalization and sensitivity to annotation quality. This paper proposes SIMS—the first supervision-free, self-improving model guidance framework—that dynamically aligns large language models with human preferences during inference via an iterative self-enhancement loop, autonomously generating and refining contrastive examples without external supervision. SIMS innovatively integrates prompt ranking with context-aware contrastive sampling, substantially improving adaptability and scenario specificity. Extensive experiments across multiple models and benchmarks demonstrate that SIMS significantly outperforms existing approaches in guidance effectiveness, robustness, and zero-annotation adaptability. It establishes a novel paradigm for unsupervised model alignment.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Diffusion Models for Vision

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Model steering represents a powerful technique that dynamically aligns large language models (LLMs) with human preferences during inference. However, conventional model-steering methods rely heavily on externally annotated data, not only limiting their adaptability to varying contexts but also tethering their effectiveness to annotation quality. In this paper, we present SIMS, the first self-improving model-steering framework that operates without relying on external supervision. At its core, SIMS autonomously generates and refines contrastive samples through iterative self-improvement cycles, enabling adaptive, context-specific steering. Additionally, SIMS employs novel strategies, including prompt ranking and contrast sampling, to further enhance steering efficacy. Extensive evaluation across diverse LLMs and benchmarks demonstrates that SIMS substantially outperforms existing methods in steering effectiveness and adaptability, highlighting self-improving model steering as a promising direction for future research on inference-time LLM alignment.

Problem

• Dynamic alignment of LLMs with human preferences
• Reducing reliance on external annotated data
• Enhancing steering effectiveness and adaptability

Innovation

• Self-improving model-steering framework without external supervision
• Autonomously generates and refines contrastive samples iteratively
• Employs prompt ranking and contrast sampling for efficacy