NumColBERT: Non-Intrusive Numeracy Injection for Late-Interaction Retrieval Models

2026-05-11

AI Summary

This work addresses the challenge that existing dense retrieval models often require separating textual and numeric scoring components when handling queries with numerical conditions, leading to deployment complexity and high latency. The authors propose NumColBERT, which enhances ColBERT’s numerical understanding without altering its architecture or indexing pipeline. By introducing a numeric gating mechanism to distinguish salient from neutral numerical values and a contrastive learning objective that aligns magnitude, units, and conditional semantics, NumColBERT preserves the original MaxSim scoring and vector indexing for efficient, low-latency end-to-end retrieval. Experimental results demonstrate that NumColBERT significantly outperforms standard fine-tuning baselines and matches or exceeds state-of-the-art methods that rely on separated scoring, offering both high performance and deployment simplicity.

Abstract

This study addresses the challenge of improving dense retrieval performance for queries containing numerical conditions, such as ``companies with more than one billion dollars in R&D expenditure.'' Although recent research has shown that standard models struggle with numeric information in domains such as finance, e-commerce, and medicine, existing solutions typically decompose queries into textual and numerical components and score them separately. These approaches modify late-interaction retrieval models such as ColBERT and introduce challenges in deployment, latency, and maintainability. To overcome these limitations, we propose NumColBERT, an inference-time non-intrusive method that enhances numerically conditioned retrieval while preserving the original late-interaction mechanism. Because NumColBERT retains the standard ColBERT indexing and MaxSim scoring pipeline, existing optimizations and ecosystem components can be reused directly, facilitating practical deployment. NumColBERT introduces a Numerical Gating Mechanism and a Numerical Contrastive Learning objective to enable numerical conditions to contribute more effectively within standard ColBERT scoring. The gating mechanism amplifies tokens carrying critical numerical constraints while suppressing context-neutral numerical mentions, and the contrastive objective shapes the embedding space to reflect numerical magnitudes, units, and conditions. Experimental results show that NumColBERT substantially outperforms standard fine-tuning baselines and achieves accuracy comparable to or better than prior approaches relying on separate textual and numerical scoring. These findings demonstrate the feasibility of numerically conditioned retrieval with a non-intrusive inference pipeline and present a maintainable solution for real-world deployment.

Problem

• numerical retrieval
• late-interaction models
• dense retrieval
• non-intrusive method
• numeracy understanding

Innovation

• NumColBERT
• numerical retrieval
• late-interaction models
• non-intrusive inference
• contrastive learning