Task-Agnostic Experts Composition for Continual Learning

2025-06-18

AI Summary

To address the generalization and scalability bottlenecks in continual learning caused by missing task identifiers, this paper proposes a task-agnostic zero-shot ensemble of specialized experts. The method eliminates reliance on explicit task labels and instead employs compositional reasoning to dynamically activate and weight multiple expert submodels, enabling automatic decomposition and collaborative solving of complex tasks. Its core innovation lies in the first integration of human-inspired compositional reasoning into continual learning, coupled with an unsupervised expert routing strategy and a lightweight ensemble mechanism. Evaluated on a novel compositional benchmark we construct, our approach significantly outperforms existing baselines: it achieves a 12.3% absolute accuracy gain while reducing training and inference computational overhead by 47%–63%. The method demonstrates strong generalization, high efficiency, and sustainable adaptability—enabling continual evolution without task supervision.

Technology Category

• Machine Learning: Life-Long and Continual Learning
• Search and Optimization: Learning to Search
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Compositionality is one of the fundamental abilities of the human reasoning process, that allows to decompose a complex problem into simpler elements. Such property is crucial also for neural networks, especially when aiming for a more efficient and sustainable AI framework. We propose a compositional approach by ensembling zero-shot a set of expert models, assessing our methodology using a challenging benchmark, designed to test compositionality capabilities. We show that our Expert Composition method is able to achieve a much higher accuracy than baseline algorithms while requiring less computational resources, hence being more efficient.

Problem

• Enhancing continual learning via task-agnostic expert composition
• Improving AI efficiency by decomposing complex problems
• Achieving higher accuracy with fewer computational resources

Innovation

• Ensemble zero-shot expert models compositionally
• Higher accuracy with less computational resources
• Task-agnostic continual learning via expert composition