An Empirical study on LLM-based Log Retrieval for Software Engineering Metadata Management

2025-06-13

AI Summary

To address the challenges of scene retrieval from massive heterogeneous autonomous driving system (ADS) test logs—namely, low retrieval accuracy, heavy reliance on domain- and database-specific expertise, and poor result verifiability—this paper proposes a natural language–driven cross-modal (log + video) joint retrieval framework. Methodologically, it integrates temporal signal modeling with multimodal alignment to construct a scene distance graph and introduces a relative gap metric for interpretable assessment. Leveraging large language models (LLMs) for query understanding, the framework encapsulates retrieval as an API service and incorporates video frame visualization for intuitive verification. Experiments on an open industrial dataset demonstrate a 32% improvement in retrieval accuracy and a 67% reduction in manual validation time. The approach significantly reduces dependence on expert prior knowledge and, for the first time, enables high-accuracy, interpretable, and easily verifiable log-video co-retrieval.

Technology Category

• Computer Vision: Image and Video Retrieval
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Developing autonomous driving systems (ADSs) involves generating and storing extensive log data from test drives, which is essential for verification, research, and simulation. However, these high-frequency logs, recorded over varying durations, pose challenges for developers attempting to locate specific driving scenarios. This difficulty arises due to the wide range of signals representing various vehicle components and driving conditions, as well as unfamiliarity of some developers' with the detailed meaning of these signals. Traditional SQL-based querying exacerbates this challenge by demanding both domain expertise and database knowledge, often yielding results that are difficult to verify for accuracy. This paper introduces a Large Language Model (LLM)-supported approach that combines signal log data with video recordings from test drives, enabling natural language based scenario searches while reducing the need for specialized knowledge. By leveraging scenario distance graphs and relative gap indicators, it provides quantifiable metrics to evaluate the reliability of query results. The method is implemented as an API for efficient database querying and retrieval of relevant records, paired with video frames for intuitive visualization. Evaluation on an open industrial dataset demonstrates improved efficiency and reliability in scenario retrieval, eliminating dependency on a single data source and conventional SQL.

Problem

• Challenges in locating specific driving scenarios in extensive log data
• Difficulty due to diverse signals and lack of domain expertise
• Traditional SQL-based querying requires specialized knowledge and is hard to verify

Innovation

• LLM-supported natural language scenario searches
• Scenario distance graphs for reliability metrics
• API for efficient querying and video visualization