CONCUR: A Framework for Continual Constrained and Unconstrained Routing

2025-12-10

AI Summary

Existing routing methods rely on monolithic models jointly trained over all strategies, necessitating full retraining upon adding new strategies—resulting in poor generalization and suboptimal decisions due to homogeneous input representations. This paper proposes a continual routing framework tailored to the heterogeneous complexity of AI tasks, supporting both budget-constrained and unconstrained routing modes. Its core contributions are: (1) a modular continual architecture that trains lightweight, strategy-specific predictors, enabling zero-shot integration of new strategies; (2) multi-perspective representation fusion, jointly encoding semantic, structural, and computational features of both tasks and strategies; and (3) a constraint-aware dynamic decision mechanism. Experiments demonstrate consistent superiority over single-strategy baselines and state-of-the-art routing methods across in-distribution/out-of-distribution settings and knowledge-intensive versus reasoning-intensive tasks—achieving higher end-to-end accuracy, lower inference cost, and up to 62% reduction in training overhead under continual learning.

Technology Category

• Planning, Routing, and Scheduling: Learning for Planning and Scheduling
• Machine Learning: Life-Long and Continual Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

AI tasks differ in complexity and are best addressed with different computation strategies (e.g., combinations of models and decoding methods). Hence, an effective routing system that maps tasks to the appropriate strategies is crucial. Most prior methods build the routing framework by training a single model across all strategies, which demands full retraining whenever new strategies appear and leads to high overhead. Attempts at such continual routing, however, often face difficulties with generalization. Prior models also typically use a single input representation, limiting their ability to capture the full complexity of the routing problem and leading to sub-optimal routing decisions. To address these gaps, we propose CONCUR, a continual routing framework that supports both constrained and unconstrained routing (i.e., routing with or without a budget). Our modular design trains a separate predictor model for each strategy, enabling seamless incorporation of new strategies with low additional training cost. Our predictors also leverage multiple representations of both tasks and computation strategies to better capture overall problem complexity. Experiments on both in-distribution and out-of-distribution, knowledge- and reasoning-intensive tasks show that our method outperforms the best single strategy and strong existing routing techniques with higher end-to-end accuracy and lower inference cost in both continual and non-continual settings, while also reducing training cost in the continual setting.

Problem

• Develops a continual routing framework for AI tasks
• Addresses generalization and overhead in strategy adaptation
• Enhances routing decisions with multiple input representations

Innovation

• Modular design trains separate predictor per strategy
• Leverages multiple representations of tasks and strategies
• Enables low-cost continual routing with new strategies