Sequentially-Controlled Interactive Multi-Particle Flow-Maps for Online Feedback-Driven Search

2026-07-01

AI Summary

This work addresses the challenge of aligning generative models with human preferences that are initially unknown and revealed only through sequential feedback, where existing methods often struggle to balance global exploration with efficient alignment, leading to suboptimal convergence or reward over-optimization. To overcome this, the paper proposes the Interactive Multi-Particle Flow Map (IMPFM) framework, which integrates sequential Monte Carlo, manifold mapping, and multi-particle interaction dynamics to construct an adaptive sampling mechanism under a KL-tilted objective. IMPFM introduces flow-map-based posterior sample sharing and a Feynman–Kac corrector to effectively preserve particle diversity, mitigate weight degeneracy, and prevent mode collapse. Empirical results demonstrate that IMPFM significantly outperforms baseline approaches across diverse tasks, achieving superior sample efficiency, enhanced global exploration, and improved accuracy in aligning with heterogeneous preferences.

Abstract

While generative models have enabled training-free reward alignment, current methods typically excel in local exploration within narrow regions of the underlying distribution. These approaches struggle when preferences are unknown a priori and only revealed through sequential feedback-a scenario demanding broad exploration to uncover high-utility regions. To address this, we propose Sequentially-Controlled Interactive Multi-Particle Flow-Maps (IMPFM), a framework for sample-efficient online feedback-driven search. IMPFM progressively transports a group of interactive particles toward the target distribution, maintaining the broad coverage essential for heterogeneous preference alignment. IMPFM introduces a principled and efficient posterior sample sharing mechanism across particles powered by flow maps. By correcting individual particle drift with the collective posterior samples of the entire ensemble at each resampling step, the framework maximizes sample utility to enable global exploration while actively mitigating reward over-optimization, typical of standard control frameworks. Paired with a principled exploration-exploitation reweighting mechanism involving multi-particle interaction, this sequentially corrected multi-particle dynamics explicitly preserves structural diversity and overcomes the weight degeneracy inherent to standard SMC samplers. Crucially, we prove that the resulting sampling framework yields a multi-particle interaction-aware Feynman-Kac corrector that progressively steers the multi-particle system toward a KL-tilted target distribution, facilitating global exploration and preventing mode collapse. Extensive empirical evaluations and rigorous ablations across diverse search and alignment tasks confirm the efficacy of IMPFM over existing baselines.

Problem

• online feedback-driven search
• global exploration
• preference alignment
• sequential feedback
• mode collapse

Innovation

• multi-particle flow-maps
• online feedback-driven search
• posterior sample sharing
• exploration-exploitation reweighting
• Feynman-Kac corrector