Can Prompts Rewind Time for LLMs? Evaluating the Effectiveness of Prompted Knowledge Cutoffs

2025-09-26

AI Summary

Large language models (LLMs) suffer from pretraining data contamination in temporal forecasting tasks, leading to inflated estimates of generalization capability. To address this, we propose “prompt-driven knowledge cutoff simulation”—a novel paradigm that systematically investigates whether prompts can induce LLMs to revert to earlier knowledge cutoff points. We construct a benchmark dataset covering three knowledge types: direct factual knowledge, semantic evolution, and causal relations, and design a multidimensional forgetting evaluation framework. Experiments show that prompting effectively suppresses explicit factual recall but fails significantly in implicit causal reasoning, revealing fundamental limitations in achieving deep temporal consistency control. Our work advances rigorous time-aware evaluation standards and provides both theoretical insights and empirical baselines for controllable knowledge forgetting.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Natural Language Processing: (Large) Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large Language Models (LLMs) are widely used for temporal prediction, but their reliance on pretraining data raises contamination concerns, as accurate predictions on pre-cutoff test data may reflect memorization rather than reasoning, leading to an overestimation of their generalization capability. With the recent emergence of prompting-based unlearning techniques, a natural question arises: Can LLMs be prompted to simulate an earlier knowledge cutoff? In this work, we investigate the capability of prompting to simulate earlier knowledge cutoff in LLMs. We construct three evaluation datasets to assess the extent to which LLMs can forget (1) direct factual knowledge, (2) semantic shifts, and (3) causally related knowledge. Results demonstrate that while prompt-based simulated knowledge cutoffs show effectiveness when directly queried with the information after that date, they struggle to induce forgetting when the forgotten content is not directly asked but causally related to the query. These findings highlight the need for more rigorous evaluation settings when applying LLMs for temporal prediction tasks. The full dataset and evaluation code are available at https://github.com/gxx27/time_unlearn.

Problem

• Evaluating prompt-based simulation of earlier knowledge cutoffs in LLMs
• Assessing LLMs' ability to forget factual and causally related knowledge
• Investigating contamination risks in temporal prediction tasks using LLMs

Innovation

• Prompted unlearning simulates earlier knowledge cutoffs
• Evaluates forgetting of factual, semantic, and causal knowledge
• Struggles with indirect causal forgetting despite direct effectiveness