Federated In-Context Learning: Iterative Refinement for Improved Answer Quality

2025-06-09

AI Summary

In privacy-sensitive question-answering (QA) tasks, in-context learning (ICL) suffers from scarce high-quality demonstration examples due to data privacy constraints, prohibitive annotation costs, and distributional shifts; meanwhile, existing federated learning approaches incur excessive communication overhead and underutilize local client data. Method: We propose Federated In-Context Learning (Fed-ICL), the first parameter-free distributed ICL framework that jointly optimizes prompts and responses across clients and server—without transmitting model parameters—via iterative collaborative refinement. Contribution/Results: Fed-ICL theoretically guarantees convergence and achieves answer quality on par with centralized ICL on standard QA benchmarks, while reducing total communication cost by over 60%. It simultaneously enhances privacy preservation and system efficiency, establishing a new trade-off frontier for privacy-aware, resource-efficient ICL in decentralized settings.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Natural Language Processing: Question Answering
• Search and Optimization: Distributed Search

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

For question-answering (QA) tasks, in-context learning (ICL) enables language models to generate responses without modifying their parameters by leveraging examples provided in the input. However, the effectiveness of ICL heavily depends on the availability of high-quality examples, which are often scarce due to data privacy constraints, annotation costs, and distribution disparities. A natural solution is to utilize examples stored on client devices, but existing approaches either require transmitting model parameters - incurring significant communication overhead - or fail to fully exploit local datasets, limiting their effectiveness. To address these challenges, we propose Federated In-Context Learning (Fed-ICL), a general framework that enhances ICL through an iterative, collaborative process. Fed-ICL progressively refines responses by leveraging multi-round interactions between clients and a central server, improving answer quality without the need to transmit model parameters. We establish theoretical guarantees for the convergence of Fed-ICL and conduct extensive experiments on standard QA benchmarks, demonstrating that our proposed approach achieves strong performance while maintaining low communication costs.

Problem

• Improves answer quality in federated question-answering tasks
• Reduces communication overhead by avoiding parameter transmission
• Addresses data scarcity and privacy constraints in federated learning

Innovation

• Federated learning without parameter transmission
• Iterative refinement via client-server interactions
• Improves answer quality with local datasets