Orthogonal Low Rank Embedding Stabilization

2025-08-10

AI Summary

In recommender systems, model retraining induces embedding space drift—degrading stability of downstream tasks such as similarity search and cold-start inference. This work proposes a lightweight, lossless embedding stabilization method that maps embeddings from successive training rounds into a unified, standardized space without altering the model architecture or training objective. Specifically, it applies an orthogonal low-rank transformation, combining efficient truncated SVD with orthogonal Procrustes alignment. The method strictly preserves dot-product invariance and inference efficiency, supports reversible alignment, and is plug-and-play compatible. Experiments demonstrate that it substantially suppresses embedding fluctuations—reducing average drift by 72%—while significantly improving downstream task stability. Crucially, it fully retains the original model’s prediction accuracy, ensuring no compromise on recommendation quality.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Data Mining & Knowledge Management: Recommender Systems
• Natural Language Processing: Safety and Robustness

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

The instability of embedding spaces across model retraining cycles presents significant challenges to downstream applications using user or item embeddings derived from recommendation systems as input features. This paper introduces a novel orthogonal low-rank transformation methodology designed to stabilize the user/item embedding space, ensuring consistent embedding dimensions across retraining sessions. Our approach leverages a combination of efficient low-rank singular value decomposition and orthogonal Procrustes transformation to map embeddings into a standardized space. This transformation is computationally efficient, lossless, and lightweight, preserving the dot product and inference quality while reducing operational burdens. Unlike existing methods that modify training objectives or embedding structures, our approach maintains the integrity of the primary model application and can be seamlessly integrated with other stabilization techniques.

Problem

• Stabilize embedding spaces across model retraining cycles
• Ensure consistent embedding dimensions during retraining
• Preserve dot product and inference quality efficiently

Innovation

• Orthogonal low-rank transformation stabilizes embeddings
• Uses SVD and Procrustes for standardized mapping
• Preserves dot product and inference quality