FACTTRACK: Time-Aware World State Tracking in Story Outlines

2024-07-23

arXiv.org

AI Summary

To address the challenge of dynamically maintaining factual consistency in story outline generation, this paper proposes FACTTRACK—a temporal-aware world state tracking framework. Methodologically, it introduces a four-stage pipeline: event decomposition, temporal validity interval inference, contradiction detection, and state update—marking the first effort to model atomic facts as dynamic entities annotated with time-validity intervals, thereby enabling lifecycle management and cross-event correction. Technically, it integrates LLaMA2-7B-Chat or GPT-4 for reasoning, employs structured world state representation, atomic-fact-oriented decomposition, and an interval-logic-based conflict detection algorithm. Experiments demonstrate that the LLaMA2-7B-Chat variant of FACTTRACK significantly outperforms comparable LLaMA2-based baselines and matches GPT-4 baseline performance; the GPT-4 variant substantially surpasses its counterpart. Results on story outline contradiction detection validate the effectiveness and advancement of temporal modeling in factual consistency maintenance.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Humans and AI: Game Design — Procedural Content Generation & Storytelling
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

While accurately detecting and correcting factual contradictions in language model outputs has become increasingly important as their capabilities improve, doing so is highly challenging. We propose a novel method, FACTTRACK, for tracking atomic facts and addressing factual contradictions. Crucially, FACTTRACK also maintains time-aware validity intervals for each fact, allowing for change over time. At a high level, FACTTRACK consists of a four-step pipeline to update a world state data structure for each new event: (1) decompose the event into directional atomic facts; (2) determine the validity interval of each atomic fact using the world state; (3) detect contradictions with existing facts in the world state; and finally (4) add new facts to the world state and update existing atomic facts. When we apply FACTTRACK to contradiction detection on structured story outlines, we find that FACTTRACK using LLaMA2-7B-Chat substantially outperforms a fair baseline using LLaMA2-7B-Chat, and achieves performance comparable to a GPT4 baseline. Moreover, when using GPT4, FACTTRACK significantly outperforms the GPT4 baseline.

Problem

• Detecting and correcting factual contradictions in language model outputs
• Tracking atomic facts with time-aware validity intervals
• Maintaining world state consistency in evolving story outlines

Innovation

• Time-aware validity intervals for facts
• Four-step pipeline for world state updates
• Decomposing events into directional atomic facts