Aligning Compound AI Systems via System-level DPO

2025-02-24

AI Summary

Addressing the alignment challenge in composite AI systems—such as LLM-based agents collaborating with diffusion models—where components are non-differentiable and system-level human preferences cannot be decomposed into component-level signals, this paper proposes System-level Direct Preference Optimization (SysDPO). SysDPO models the composite system as a directed acyclic graph (DAG) and defines preferences at the system level, introducing a differentiable surrogate objective to enable end-to-end joint alignment of non-differentiable, multi-component architectures. Unlike conventional DPO—designed for monolithic models—SysDPO supports heterogeneous component co-training. Experiments demonstrate significant improvements in output consistency and alignment with human preferences across multi-step reasoning and generative tasks. SysDPO establishes the first scalable, generalizable system-level alignment paradigm for composite AI systems. (136 words)

Technology Category

• Multiagent Systems: Adversarial Agents
• Humans and AI: Learning Human Values and Preferences
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Agentic search

Abstract

Compound AI systems, comprising multiple interacting components such as LLM agents and external tools, demonstrate state-of-the-art results across diverse tasks. It is hence crucial to align components within the system to produce consistent results that match human expectations. However, conventional alignment methods, such as Direct Preference Optimization (DPO), are not directly applicable to compound AI systems. These challenges include the non-differentiable interactions between components, making end-to-end gradient optimization infeasible. Additionally, system-level preferences cannot be directly translated into component-level preferences, further complicating alignment. We address the issues by formulating compound AI systems as Directed Acyclic Graphs (DAGs), capturing the connections between agents and the data generation processes. We propose a system-level DPO (SysDPO) to jointly align compound systems by adapting the DPO to operate on these DAGs. We study the joint alignment of an LLM and a diffusion model to demonstrate the effectiveness of our approach. Our exploration provides insights into the alignment of compound AI systems and lays a foundation for future advancements.

Problem

• Align components in compound AI systems.
• Develop system-level DPO for non-differentiable interactions.
• Translate system-level preferences into component-level preferences.

Innovation

• System-level DPO for alignment
• DAGs model compound AI systems
• Jointly align LLM and diffusion