LLM-Enhanced Reinforcement Learning for Time Series Anomaly Detection

2026-01-05

arXiv.org

AI Summary

This work addresses the challenges of sparse annotations, complex temporal patterns, and high labeling costs in time series anomaly detection by proposing a novel approach that integrates semantic priors from large language models (LLMs) with reinforcement learning. The method leverages LLM-generated semantic rewards to guide an LSTM-based policy exploration, while reconstruction errors from a variational autoencoder (VAE) provide unsupervised anomaly signals. To efficiently expand limited labeled data, it incorporates active learning and label propagation mechanisms. Innovatively, semantic knowledge from the LLM is employed for reward shaping, complemented by a VAE-driven dynamic reward scaling strategy. Evaluated on the Yahoo-A1 and SMD benchmarks, the proposed method achieves state-of-the-art detection accuracy under extremely low annotation budgets, significantly outperforming existing techniques.

Technology Category

• Machine Learning: Time-Series/Data Streams
• Data Mining & Knowledge Management: Anomaly/Outlier Detection
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Detecting anomalies in time series data is crucial for finance, healthcare, sensor networks, and industrial monitoring applications. However, time series anomaly detection often suffers from sparse labels, complex temporal patterns, and costly expert annotation. We propose a unified framework that integrates Large Language Model (LLM)-based potential functions for reward shaping with Reinforcement Learning (RL), Variational Autoencoder (VAE)-enhanced dynamic reward scaling, and active learning with label propagation. An LSTM-based RL agent leverages LLM-derived semantic rewards to guide exploration, while VAE reconstruction errors add unsupervised anomaly signals. Active learning selects the most uncertain samples, and label propagation efficiently expands labeled data. Evaluations on Yahoo-A1 and SMD benchmarks demonstrate that our method achieves state-of-the-art detection accuracy under limited labeling budgets and operates effectively in data-constrained settings. This study highlights the promise of combining LLMs with RL and advanced unsupervised techniques for robust, scalable anomaly detection in real-world applications.

Problem

• time series anomaly detection
• sparse labels
• complex temporal patterns
• expert annotation cost

Innovation

• LLM-enhanced reinforcement learning
• reward shaping
• VAE-based anomaly detection
• active learning
• label propagation