Quantum-inspired Embeddings Projection and Similarity Metrics for Representation Learning

2025-01-08

AI Summary

This work addresses two key challenges in representation learning: redundancy in projection heads and mismatch between embedding similarity and downstream task metrics. We propose a quantum-inspired framework that maps classical embeddings to low-entanglement quantum states and employs parameterized quantum circuits to achieve efficient, geometry-preserving compression in Hilbert space. By co-designing quantum-state encoding with circuit-based projection heads, our method preserves the intrinsic similarity structure of embeddings while reducing parameter count by 32×. Integrated seamlessly into BERT, it matches state-of-the-art performance on TREC 2019/2020 information retrieval benchmarks and significantly outperforms baselines under zero-shot and few-shot settings. To our knowledge, this is the first approach to jointly optimize quantum-state mapping and lightweight projection heads, establishing a novel paradigm for representation compression in large language models and cross-modal alignment.

Technology Category

• Machine Learning: Quantum Machine Learning
• Search and Optimization: Learning to Search
• Computer Vision: Representation Learning for Vision

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Over the last decade, representation learning, which embeds complex information extracted from large amounts of data into dense vector spaces, has emerged as a key technique in machine learning. Among other applications, it has been a key building block for large language models and advanced computer vision systems based on contrastive learning. A core component of representation learning systems is the projection head, which maps the original embeddings into different, often compressed spaces, while preserving the similarity relationship between vectors. In this paper, we propose a quantum-inspired projection head that includes a corresponding quantum-inspired similarity metric. Specifically, we map classical embeddings onto quantum states in Hilbert space and introduce a quantum circuit-based projection head to reduce embedding dimensionality. To evaluate the effectiveness of this approach, we extended the BERT language model by integrating our projection head for embedding compression. We compared the performance of embeddings, which were compressed using our quantum-inspired projection head, with those compressed using a classical projection head on information retrieval tasks using the TREC 2019 and TREC 2020 Deep Learning benchmarks. The results demonstrate that our quantum-inspired method achieves competitive performance relative to the classical method while utilizing 32 times fewer parameters. Furthermore, when trained from scratch, it notably excels, particularly on smaller datasets. This work not only highlights the effectiveness of the quantum-inspired approach but also emphasizes the utility of efficient, ad hoc low-entanglement circuit simulations within neural networks as a powerful quantum-inspired technique.

Problem

• Quantum-inspired Learning
• Representation Learning
• Large Language Models

Innovation

• Quantum-Inspired Embedding
• Similarity Computation
• Parameter Efficiency