MicLog: Towards Accurate and Efficient LLM-based Log Parsing via Progressive Meta In-Context Learning

2026-01-11

arXiv.org

AI Summary

Existing log parsing methods suffer from limited semantic generalization and insufficient domain coverage, while large language model (LLM)-based approaches are often hindered by suboptimal in-context learning utilization and low query efficiency. To address these limitations, this work proposes MicLog, a novel framework that integrates meta-learning with in-context learning to establish a progressive meta in-context learning (ProgMeta-ICL) paradigm spanning from zero-shot to K-shot settings. MicLog further introduces weighted DBSCAN candidate sampling, enhanced BM25-based example selection, and a multi-level dynamic template-matching cache mechanism. Evaluated on Loghub-2.0 using the compact open-source LLM Qwen-2.5-3B, MicLog achieves a 10.3% improvement in parsing accuracy and a 42.4% reduction in processing time compared to existing methods.

Technology Category

• Search and Optimization: Metareasoning and Metaheuristics
• Machine Learning: Statistical Relational/Logic Learning
• Natural Language Processing: Syntax — Tagging, Chunking & Parsing

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Web traffic and log analysis

Abstract

Log parsing converts semi-structured logs into structured templates, forming a critical foundation for downstream analysis. Traditional syntax and semantic-based parsers often struggle with semantic variations in evolving logs and data scarcity stemming from their limited domain coverage. Recent large language model (LLM)-based parsers leverage in-context learning (ICL) to extract semantics from examples, demonstrating superior accuracy. However, LLM-based parsers face two main challenges: 1) underutilization of ICL capabilities, particularly in dynamic example selection and cross-domain generalization, leading to inconsistent performance; 2) time-consuming and costly LLM querying. To address these challenges, we present MicLog, the first progressive meta in-context learning (ProgMeta-ICL) log parsing framework that combines meta-learning with ICL on small open-source LLMs (i.e., Qwen-2.5-3B). Specifically, MicLog: i) enhances LLMs'ICL capability through a zero-shot to k-shot ProgMeta-ICL paradigm, employing weighted DBSCAN candidate sampling and enhanced BM25 demonstration selection; ii) accelerates parsing via a multi-level pre-query cache that dynamically matches and refines recently parsed templates. Evaluated on Loghub-2.0, MicLog achieves 10.3% higher parsing accuracy than the state-of-the-art parser while reducing parsing time by 42.4%.

Problem

• log parsing
• in-context learning
• large language models
• cross-domain generalization
• query efficiency

Innovation

• progressive meta in-context learning
• log parsing
• small LLMs
• multi-level cache
• BM25 demonstration selection