Fed-SE: Federated Self-Evolution for Privacy-Constrained Multi-Environment LLM Agents

2025-12-09

AI Summary

Existing federated learning approaches suffer from gradient conflicts and unstable global optimization in open-ended, privacy-constrained scenarios where large language model (LLM) agents must self-evolve across heterogeneous environments. Method: We propose a decentralized federated evolution framework featuring a “local evolution–global aggregation” paradigm: (i) lightweight local policy evolution via parameter-efficient fine-tuning (PEFT); (ii) trajectory-level high-return sample filtering to enhance policy quality; and (iii) a novel low-rank subspace gradient aggregation mechanism that decouples environment-specific dynamics and mitigates negative transfer. Contribution/Results: Experiments across five heterogeneous environments demonstrate an average 18% improvement in task success rate over state-of-the-art federated baselines. To our knowledge, this is the first work achieving robust, privacy-preserving cross-environment self-evolution of LLM agents.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Multiagent Systems: Adversarial Agents
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

LLM agents are widely deployed in complex interactive tasks, yet privacy constraints often preclude centralized optimization and co-evolution across dynamic environments. While Federated Learning (FL) has proven effective on static datasets, its extension to the open-ended self-evolution of agents remains underexplored. Directly applying standard FL is challenging: heterogeneous tasks and sparse, trajectory-level rewards introduce severe gradient conflicts, destabilizing the global optimization process. To bridge this gap, we propose Fed-SE, a Federated Self-Evolution framework for LLM agents. Fed-SE establishes a local evolution-global aggregation paradigm. Locally, agents employ parameter-efficient fine-tuning on filtered, high-return trajectories to achieve stable gradient updates. Globally, Fed-SE aggregates updates within a low-rank subspace that disentangles environment-specific dynamics, effectively reducing negative transfer across clients. Experiments across five heterogeneous environments demonstrate that Fed-SE improves average task success rates by approximately 18% over federated baselines, validating its effectiveness in robust cross-environment knowledge transfer in privacy-constrained deployments.

Problem

• Federated self-evolution for privacy-constrained multi-environment LLM agents
• Address gradient conflicts from heterogeneous tasks and sparse rewards
• Enable robust cross-environment knowledge transfer without centralized data

Innovation

• Local evolution uses filtered high-return trajectories for stable fine-tuning
• Global aggregation disentangles dynamics in a low-rank subspace
• Reduces negative transfer across clients in privacy-constrained environments