When Does Multi-Agent RL Improve LLM Workflows? Workflow, Scale, and Policy-Sharing Tradeoffs

2026-05-22

AI Summary

This study addresses the poor training stability of end-to-end reinforcement learning in multi-agent large language model (LLM) workflows, where performance is governed by an unclear coupling among workflow architecture, task type, and model scale. The authors systematically investigate training dynamics under shared and isolated policy settings across three canonical workflows—Eval-Opt, Voting, and Orch-Workers—on mathematical and code generation tasks, spanning model sizes from 0.6B to 4B parameters. Through multi-agent reinforcement learning, end-to-end training, and role-level gradient analysis, they reveal that policy sharing does not universally enhance stability; instead, it propagates training pressure along distinct pathways, leading to failure modes contingent on workflow topology and task characteristics. While isolated policies achieve higher peak performance, they are prone to collapse, and overall gains are jointly determined by the interplay of workflow design, task domain, and model scale.

Abstract

Multi-agent LLM workflows route inference through specialized roles to lift end-task accuracy, but jointly training those roles with reinforcement learning is unstable in ways that are poorly understood. We study when end-to-end RL training of multi-agent LLM workflows improves over their base models, comparing Shared-Policy training, where all roles update one policy, with Isolated-Policy training, where each role has its own parameters. Our experimental matrix spans Eval-Opt, Voting, and Orch-Workers workflows, math and code tasks, and three model scales (0.6B, 1.7B, 4B). We find that multi-agent RL usually improves over base models, but gains depend jointly on workflow, task, and scale, not on policy sharing alone. Isolated-Policy tends to reach higher peak accuracy yet more often falls off a terminal accuracy cliff, while Shared-Policy training does not eliminate failure; it redistributes failure into qualitatively different patterns. We then explain the strongest of these patterns through role-level gradient dynamics induced by workflow topology and policy routing: under Isolated-Policy, parallel same-role agents on shared prompts amplify per-role gradients and drive terminal degradation in Voting and Orch-Workers workflows; under Shared-Policy, asymmetric per-step gradient mass causes the shared policy to be captured by the dominant role, producing different failure signatures by task and workflow. Together, the empirical map and its underlying mechanisms show that policy sharing routes training pressure through different channels rather than offering uniform stability, making it a design choice with workflow- and task-conditional tradeoffs.

Problem

• Multi-Agent RL
• LLM Workflows
• Policy Sharing
• Training Stability
• Gradient Dynamics

Innovation

• multi-agent reinforcement learning
• LLM workflows
• policy sharing
• gradient dynamics
• training stability