Policy Regret for Embedding Model Routing: Contextual Bandits with Low-Rank Experts

2026-06-12

AI Summary

This work addresses the dynamic routing problem among multiple embedding models under realistic settings involving adversarial queries, bandit feedback, and limited model observability. The problem is formulated as an adversarial contextual linear bandit with a low-rank expert structure. To tackle this, the authors introduce, for the first time, a log-quadratic policy class that balances expressiveness and learnability, and develop HPG (Hypentropy Policy Gradient)—a parameter-free policy gradient algorithm that adapts automatically to the unknown low-rank structure without requiring hyperparameter tuning. The algorithm achieves a policy regret bound of $\tilde{\mathcal{O}}(s \sqrt{M T})$, where $s$ denotes the intrinsic rank of the experts, $M$ is the number of models, and $T$ is the number of rounds, thereby offering both theoretical optimality and computational efficiency.

Abstract

Modern recommendation systems increasingly rely on dynamically routing diverse queries to multiple embedding models. Despite its practical significance, this problem remains poorly understood under realistic conditions like adversarial queries, bandit feedback, and limited observability of models. We formalize embedding model routing as an adversarial contextual linear bandit with low-rank experts, where contexts are queries, actions are items, and experts are the embedding models working on low-rank latent representation spaces. We first establish that standard regret notions suffer from structural misspecification or statistical intractability, and we identify a log-quadratic policy class that is expressive enough to capture query-dependent model routing, yet structured enough to allow efficient online learning. Second, we propose a policy gradient algorithm called Hypentropy Policy Gradient (HPG). It provably adapts to the unknown low-rank structure under incomplete information and attains $\tilde{\mathcal O}(s\sqrt{M T})$ linearized policy regret -- where $s, M$, and $T$ are the intrinsic rank of the experts, the number of models, and the number of rounds -- thus avoiding a curse of dimensionality. Finally, we also provide an computationally efficient and parameter-free implementation of HPG.

Problem

• embedding model routing
• contextual bandits
• low-rank experts
• adversarial queries
• bandit feedback

Innovation

• low-rank experts
• contextual bandits
• policy regret
• embedding model routing
• policy gradient