Towards Temporal Knowledge-Base Creation for Fine-Grained Opinion Analysis with Language Models

2025-09-02

AI Summary

Existing time-series opinion analysis suffers from a lack of fine-grained, temporally aligned annotations, hindering downstream tasks such as forecasting and trend analysis. To address this, we propose a declarative large language model (LLM) annotation framework that integrates classical opinion mining structures—sentiment and aspect-opinion pairs—directly into the LLM pipeline, enabling fully automated generation of high-quality, temporally aligned structured opinion data without handcrafted prompts. Our approach innovatively incorporates label-level consistency evaluation and time-aware modeling to ensure temporal coherence and structural fidelity. This yields the first knowledge base explicitly designed for time-series opinion understanding. The resulting knowledge base supports diverse applications, including retrieval-augmented generation (RAG), temporal question answering, and timeline summarization. On a human-validated test set, it achieves high inter-annotator agreement (Cohen’s κ = 0.89) and accuracy (F1 = 0.92), significantly enhancing both the practical utility and scalability of time-series opinion mining.

Technology Category

• Data Mining & Knowledge Management: Linked Open Data, Knowledge Graphs & KB Completion
• Natural Language Processing: (Large) Language Models
• Machine Learning: Time-Series/Data Streams

Application Category

• Web Mining and Content Analysis: Sentiment analysis and opinion mining
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

We propose a scalable method for constructing a temporal opinion knowledge base with large language models (LLMs) as automated annotators. Despite the demonstrated utility of time-series opinion analysis of text for downstream applications such as forecasting and trend analysis, existing methodologies underexploit this potential due to the absence of temporally grounded fine-grained annotations. Our approach addresses this gap by integrating well-established opinion mining formulations into a declarative LLM annotation pipeline, enabling structured opinion extraction without manual prompt engineering. We define three data models grounded in sentiment and opinion mining literature, serving as schemas for structured representation. We perform rigorous quantitative evaluation of our pipeline using human-annotated test samples. We carry out the final annotations using two separate LLMs, and inter-annotator agreement is computed label-wise across the fine-grained opinion dimensions, analogous to human annotation protocols. The resulting knowledge base encapsulates time-aligned, structured opinions and is compatible with applications in Retrieval-Augmented Generation (RAG), temporal question answering, and timeline summarisation.

Problem

• Constructing temporal opinion knowledge base with LLMs
• Addressing lack of fine-grained temporal opinion annotations
• Enabling structured opinion extraction without manual engineering

Innovation

• LLMs as automated annotators for scalable knowledge base creation
• Declarative pipeline integrating opinion mining without manual engineering
• Temporally grounded structured extraction for RAG and timeline applications