Adapting Large Language Models for Parameter-Efficient Log Anomaly Detection

2025-03-11

AI Summary

This work addresses two key limitations in applying large language models (LLMs) to log anomaly detection (LAD): restricted practical applicability and prohibitive computational cost of full-parameter fine-tuning. We present the first systematic investigation of parameter-efficient fine-tuning (PEFT) for LAD, innovatively integrating LoRA and Representation Fine-tuning (ReFT) into this domain. Our approach is rigorously evaluated across diverse model architectures—RoBERTa, GPT-2, and Llama-3—and four benchmark public log datasets. Experimental results demonstrate substantial improvements in detection accuracy and robustness under low-resource and noisy conditions, alongside superior sample efficiency and cross-domain generalization capability. The proposed framework significantly reduces training overhead while maintaining strong performance, offering a reproducible, low-cost paradigm for leveraging LLMs in system log security analysis. All code is publicly released.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Log Anomaly Detection (LAD) seeks to identify atypical patterns in log data that are crucial to assessing the security and condition of systems. Although Large Language Models (LLMs) have shown tremendous success in various fields, the use of LLMs in enabling the detection of log anomalies is largely unexplored. This work aims to fill this gap. Due to the prohibitive costs involved in fully fine-tuning LLMs, we explore the use of parameter-efficient fine-tuning techniques (PEFTs) for adapting LLMs to LAD. To have an in-depth exploration of the potential of LLM-driven LAD, we present a comprehensive investigation of leveraging two of the most popular PEFTs -- Low-Rank Adaptation (LoRA) and Representation Fine-tuning (ReFT) -- to tap into three prominent LLMs of varying size, including RoBERTa, GPT-2, and Llama-3, for parameter-efficient LAD. Comprehensive experiments on four public log datasets are performed to reveal important insights into effective LLM-driven LAD in several key perspectives, including the efficacy of these PEFT-based LLM-driven LAD methods, their stability, sample efficiency, robustness w.r.t. unstable logs, and cross-dataset generalization. Code is available at https://github.com/mala-lab/LogADReft.

Problem

• Adapting LLMs for efficient log anomaly detection
• Exploring parameter-efficient fine-tuning techniques for LLMs
• Evaluating LLM-driven LAD methods across multiple datasets

Innovation

• Parameter-efficient fine-tuning for LLMs
• LoRA and ReFT techniques for log anomaly detection
• Comprehensive experiments on public log datasets