Bilevel Optimization of Synthetic Trajectories for Multi-Turn LLM Fine-Tuning

2026-05-23

AI Summary

This work addresses the performance degradation in multi-turn large language model (LLM) fine-tuning caused by low-quality synthetic trajectories. To mitigate this issue, the authors propose BOOST, a novel framework that introduces bilevel optimization for trajectory reweighting: the inner loop trains the LLM using continuous trajectory weights, while the outer loop optimizes a lightweight reweighting head on real validation tasks, dynamically upweighting high-quality trajectories without relying on external reward models. Theoretically, the paper establishes a PAC-Bayesian bound that characterizes a three-way trade-off among data diversity, task shift, and sample efficiency. Empirical results demonstrate that BOOST significantly outperforms existing methods on multi-turn interactive tasks, effectively amplifying the influence of high-quality synthetic data aligned with the true data distribution.

Abstract

While LLMs excel at single-turn generation, they struggle with long-horizon, multi-turn interactions. Offline reinforcement learning (RL) offers a scalable approach, yet its performance hinges on the availability and quality of multi-turn trajectory data. A common remedy is to augment training with synthetic trajectories generated by LLMs or simulators, but synthetic data is highly heterogeneous in quality, and naively treating all trajectories as equally informative can degrade performance. We propose BOOST, a bilevel optimization framework where the inner level trains the LLM on reweighted data and the outer level trains a lightweight reweighting head on held-out real validation tasks, assigning continuous trajectory-level weights without requiring an external judge. To ground this approach, we derive a PAC-Bayesian bound revealing a three-way trade-off: synthetic data increases diversity but risks task-shift, while concentrating weight on high-quality trajectories improves empirical performance at the cost of effective sample size. Empirically, our method consistently outperforms multiple baselines. Analysis reveals it upweights synthetic trajectories that align with the real data distribution and exhibit higher qualitative merit.

Problem

• bilevel optimization
• synthetic trajectories
• multi-turn LLM fine-tuning
• offline reinforcement learning
• trajectory weighting

Innovation

• bilevel optimization
• synthetic trajectories
• offline reinforcement learning
• trajectory reweighting
• PAC-Bayesian bound