Scalable Dynamic Embedding Size Search for Streaming Recommendation

2024-07-22

International Conference on Information and Knowledge Management

AI Summary

To address the unbounded embedding storage overhead caused by continuously growing users and items in streaming recommendation, this paper proposes Scalable Lightweight Embeddings (SCALL), which adaptively allocates embedding dimensions per entity under strict memory budget constraints. Its core contributions are threefold: (1) the first probabilistic distribution-based embedding size sampling mechanism, enabling precise memory control; (2) a reinforcement learning search paradigm with fixed-length state representations, supporting dynamic size generation for unseen entities; and (3) mean-pooling-based state modeling coupled with dynamic memory-aware optimization. Experiments on two public streaming recommendation datasets demonstrate that SCALL reduces storage overhead by up to 62% while improving recommendation accuracy—Recall@10 increases by 3.1–5.7%. To our knowledge, SCALL is the first method to jointly optimize memory controllability and model performance in streaming recommendation.

Technology Category

• Data Mining & Knowledge Management: Data Stream Mining
• Search and Optimization: Learning to Search
• Machine Learning: Scalability of ML Systems

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

Recommender systems typically represent users and items by learning their embeddings, which are usually set to uniform dimensions and dominate the model parameters. However, real-world recommender systems often operate in streaming recommendation scenarios, where the number of users and items continues to grow, leading to substantial storage resource consumption for these embeddings. Although a few methods attempt to mitigate this by employing embedding size search strategies to assign different embedding dimensions in streaming recommendations, they assume that the embedding size grows with the frequency of users/items, which eventually still exceeds the predefined memory budget over time. To address this issue, this paper proposes to learn Scalable Lightweight Embeddings for streaming recommendation, called SCALL, which can adaptively adjust the embedding sizes of users/items within a given memory budget over time. Specifically, we propose to sample embedding sizes from a probabilistic distribution, with the guarantee to meet any predefined memory budget. By fixing the memory budget, the proposed embedding size sampling strategy can increase and decrease the embedding sizes in accordance to the frequency of the corresponding users or items. Furthermore, we develop a reinforcement learning-based search paradigm that models each state with mean pooling to keep the length of the state vectors fixed, invariant to the changing number of users and items. As a result, the proposed method can provide embedding sizes to unseen users and items. Comprehensive empirical evaluations on two public datasets affirm the advantageous effectiveness of our proposed method.

Problem

• Dynamic embedding size adaptation within fixed memory budget
• Addressing storage growth from streaming user/item expansion
• Reinforcement learning search for scalable embedding dimensions

Innovation

• Adaptive embedding size adjustment within memory budget
• Probabilistic sampling for embedding size allocation
• Reinforcement learning with fixed-length state vectors