TeamTR: Trust-Region Fine-Tuning for Multi-Agent LLM Coordination

2026-05-01

AI Summary

This work addresses the degradation in coordination performance observed in multi-agent large language models under sequential fine-tuning, where accumulated occupancy distribution shifts can lead to results worse than single-agent baselines. To mitigate this issue, we propose TeamTR, a framework that resamples trajectories after each policy update and employs trust-region-based policy optimization with per-agent KL divergence constraints to control policy divergence. We formally characterize the cumulative occupancy shift problem and prove that intermediate occupancy evaluation reduces the error bound from quadratic to linear in the number of updates, establishing theoretical lower bounds on performance improvement per update and per training stage. Empirical results demonstrate that TeamTR outperforms both single-agent and sequential fine-tuning baselines by 7.1% on average, effectively alleviating coordination collapse while supporting a plug-and-play modular design.

Abstract

Multi-agent LLM systems have shown promise for complex reasoning, yet recent evaluations reveal they often underperform single-model baselines. We identify a structural failure mode in sequential fine-tuning of shared-context teams: updating one agent shifts the team's context distribution, and when subsequent updates are evaluated on cached rollouts, this mismatch compounds. We formalize this as the compounding occupancy shift and prove that stale-occupancy evaluation incurs a penalty that scales quadratically with the number of agents. In contrast, intermediate-occupancy evaluation reduces this to linear scaling. We propose TeamTR, a trust-region framework that resamples trajectories after each component update and enforces per-agent divergence control, yielding rigorous per-update and per-stage improvement lower bounds. Experiments show that TeamTR outperforms single-agent and sequential baselines with 7.1% on average, mitigates coordination regressions, and supports plug-and-play component replacement. Code is available at https://github.com/Yydc/TeamTR.

Problem

• multi-agent LLM coordination
• compounding occupancy shift
• sequential fine-tuning
• distribution mismatch
• context distribution shift

Innovation

• trust-region
• multi-agent LLM
• occupancy shift
• fine-tuning
• trajectory resampling