DateLogicQA: Benchmarking Temporal Biases in Large Language Models

2024-12-17

arXiv.org

AI Summary

This paper addresses the dual representation-level and logic-level biases prevalent in large language models (LLMs) for temporal reasoning. To this end, the authors propose a systematic diagnostic framework. They introduce the first fine-grained date-logic question-answering benchmark (190 questions), covering multi-format dates, temporal contexts, and diverse reasoning types. They formally define “semantic completeness” to quantify tokenization-induced bias and decouple failures originating from the representation layer versus the logical reasoning layer. Leveraging embedding-space analysis, output consistency checks, and controlled reasoning-path evaluation, they comprehensively assess mainstream LLMs. Experiments reveal critical weaknesses in date parsing, cross-format temporal alignment, and sequential logical deduction. The work establishes a novel, interpretable paradigm for evaluating and enhancing LLMs’ time-aware reasoning capabilities.

Technology Category

• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

This paper introduces DateLogicQA, a benchmark with 190 questions covering diverse date formats, temporal contexts, and reasoning types. We propose the Semantic Integrity Metric to assess tokenization quality and analyse two biases: Representation-Level Bias, affecting embeddings, and Logical-Level Bias, influencing reasoning outputs. Our findings provide a comprehensive evaluation of LLMs' capabilities and limitations in temporal reasoning, highlighting key challenges in handling temporal data accurately. The GitHub repository for our work is available at https://github.com/gagan3012/EAIS-Temporal-Bias

Problem

• Evaluating temporal biases in LLMs using DateLogicQA benchmark
• Assessing tokenization quality with Semantic Integrity Metric
• Analyzing Representation-Level and Logical-Level temporal biases

Innovation

• Introduces DateLogicQA benchmark for temporal reasoning
• Proposes Semantic Integrity Metric for tokenization
• Analyzes Representation and Logical-Level Biases