Advancing Sequential Numerical Prediction in Autoregressive Models

2025-05-19

AI Summary

Autoregressive models treat numerical sequences as discrete tokens, disregarding their inherent ordinal structure and global consistency, leading to prediction distortion. To address this, we propose the Numerical Token Integrity Loss (NTIL), the first token-level loss that incorporates an extended Earth Mover’s Distance (EMD) to explicitly model numerical ordinal relationships, while jointly optimizing sequence-level fidelity to preserve overall structural integrity—enabling dual-granularity numerical awareness. NTIL is plug-and-play, seamlessly integrating into any autoregressive decoding framework and fully compatible with LLM and multimodal LLM training paradigms. Empirical evaluation across diverse numerical forecasting tasks—including mathematical reasoning, financial time series, and scientific computing—demonstrates substantial improvements in both prediction accuracy and sequence structural consistency. These results validate NTIL’s generalizability, robustness, and practical utility for numerical sequence modeling.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Autoregressive models have become the de facto choice for sequence generation tasks, but standard approaches treat digits as independent tokens and apply cross-entropy loss, overlooking the coherent structure of numerical sequences. This paper introduces Numerical Token Integrity Loss (NTIL) to address this gap. NTIL operates at two levels: (1) token-level, where it extends the Earth Mover's Distance (EMD) to preserve ordinal relationships between numerical values, and (2) sequence-level, where it penalizes the overall discrepancy between the predicted and actual sequences. This dual approach improves numerical prediction and integrates effectively with LLMs/MLLMs. Extensive experiments show significant performance improvements with NTIL.

Problem

• Addresses autoregressive models' neglect of numerical sequence coherence
• Introduces NTIL for token and sequence-level numerical integrity
• Improves numerical prediction accuracy in LLMs/MLLMs

Innovation

• Introduces Numerical Token Integrity Loss (NTIL)
• Extends Earth Mover's Distance for ordinal preservation
• Penalizes sequence-level discrepancies in predictions