From Data to Rewards: a Bilevel Optimization Perspective on Maximum Likelihood Estimation

2025-10-08

AI Summary

This work addresses the challenge of aligning generative models in the absence of explicit reward signals. We propose a novel paradigm that automatically infers implicit reward functions from high-quality data. Methodologically, we formulate a bilevel optimization framework: the inner loop updates the generative policy via policy gradient methods, while the outer loop jointly optimizes a parameterized reward function—thereby bridging maximum likelihood estimation and reinforcement learning objectives. Theoretical analysis establishes convergence guarantees. Empirical evaluation spans tabular classification and model-based reinforcement learning tasks, demonstrating significant improvements in generalization and substantial mitigation of catastrophic forgetting. The implementation is publicly available. Our approach provides a scalable, interpretable, and unified framework for reward-free alignment learning, eliminating the need for hand-crafted reward design.

Technology Category

• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Search and Optimization: Learning to Search
• Natural Language Processing: Generation

Application Category

• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Generative models form the backbone of modern machine learning, underpinning state-of-the-art systems in text, vision, and multimodal applications. While Maximum Likelihood Estimation has traditionally served as the dominant training paradigm, recent work have highlighted its limitations, particularly in generalization and susceptibility to catastrophic forgetting compared to Reinforcement Learning techniques, such as Policy Gradient methods. However, these approaches depend on explicit reward signals, which are often unavailable in practice, leaving open the fundamental problem of how to align generative models when only high-quality datasets are accessible. In this work, we address this challenge via a Bilevel Optimization framework, where the reward function is treated as the optimization variable of an outer-level problem, while a policy gradient objective defines the inner-level. We then conduct a theoretical analysis of this optimization problem in a tractable setting and extract insights that, as we demonstrate, generalize to applications such as tabular classification and model-based reinforcement learning. We release the code at https://github.com/abenechehab/nll_to_po .

Problem

• Aligning generative models without explicit reward signals
• Overcoming Maximum Likelihood Estimation limitations via bilevel optimization
• Optimizing reward functions when only high-quality datasets exist

Innovation

• Bilevel optimization framework for reward learning
• Policy gradient objective defines inner optimization level
• Reward function treated as outer optimization variable