When Synthetic Data Hurts: On Catastrophic Forgetting in Skill Retrieval for LLM Agents

2026-09-09

AI Summary

研究解决了LLM代理在技能检索中的灾难性遗忘问题，通过使用持续学习启发的方法如嵌入锚正则化等，改善了合成数据和真实数据上的技能检索性能。

Abstract

LLM agents increasingly rely on external skills retrieved at runtime, making skill selection from large repositories a critical challenge. We present a production skill router over 34,396 skills and a large-scale study of skill retrieval using limited real supervision and synthetic data. We found that the synthetic-data fine-tuning improves in-distribution retrieval but it causes catastrophic forgetting on real and out-of-distribution (OOD) data. We evaluate several forgetting mitigation fine-tuning approaches inspired by continual learning, including embedding-anchor regularization, Learning without Forgetting (LwF), Elastic Weight Consolidation (EWC), and L2-initialization. The results show that these approaches not only retain the performance on OOD skills retrieval but also improve the retrieval on synthetic in-distribution skills by 13.98\% for 0.6B Qwen retriever and reranker. Our results provide a practical benchmark and a robust fine-tuning recipe for scarce, multi-positive supervision.

Problem

• Catastrophic Forgetting
• Synthetic Data
• Skill Retrieval
• LLM Agents
• Continual Learning

Innovation

• catastrophic forgetting
• continual learning
• synthetic data fine-tuning
• embedding-anchor regularization
• Learning without Forgetting (LwF)