GateLens: A Reasoning-Enhanced LLM Agent for Automotive Software Release Analytics

2025-03-27

AI Summary

To address low efficiency in manual analysis and insufficient accuracy and safety-controllability of large language models (LLMs) for structured tabular data reasoning in automotive software release validation, this paper proposes a two-stage interpretable reasoning framework. First, natural-language queries are formally mapped to relational algebra expressions; second, executable Python code is synthesized and optimized. The method integrates relational algebra modeling, LLM-based program synthesis, query rewriting, and execution verification—requiring no few-shot examples and enabling end-to-end traceable analysis. On benchmark evaluation, our approach achieves significantly higher F1 scores than baseline methods. In industrial deployment, it reduces analysis time by over 80% while maintaining high precision. Moreover, it supports generalization across multiple stakeholder roles and diverse validation scenarios, and has been successfully integrated into the release validation pipelines of leading automotive OEMs.

Technology Category

• Knowledge Representation and Reasoning: Automated Reasoning and Theorem Proving
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Ensuring the reliability and effectiveness of software release decisions is critical, particularly in safety-critical domains like automotive systems. Precise analysis of release validation data, often presented in tabular form, plays a pivotal role in this process. However, traditional methods that rely on manual analysis of extensive test datasets and validation metrics are prone to delays and high costs. Large Language Models (LLMs) offer a promising alternative but face challenges in analytical reasoning, contextual understanding, handling out-of-scope queries, and processing structured test data consistently; limitations that hinder their direct application in safety-critical scenarios. This paper introduces GateLens, an LLM-based tool for analyzing tabular data in the automotive domain. GateLens translates natural language queries into Relational Algebra (RA) expressions and then generates optimized Python code. It outperforms the baseline system on benchmarking datasets, achieving higher F1 scores and handling complex and ambiguous queries with greater robustness. Ablation studies confirm the critical role of the RA module, with performance dropping sharply when omitted. Industrial evaluations reveal that GateLens reduces analysis time by over 80% while maintaining high accuracy and reliability. As demonstrated by presented results, GateLens achieved high performance without relying on few-shot examples, showcasing strong generalization across various query types from diverse company roles. Insights from deploying GateLens with a partner automotive company offer practical guidance for integrating AI into critical workflows such as release validation. Results show that by automating test result analysis, GateLens enables faster, more informed, and dependable release decisions, and can thus advance software scalability and reliability in automotive systems.

Problem

• Automating analysis of automotive software release validation data
• Enhancing LLM reasoning for tabular data in safety-critical domains
• Reducing delays and costs in manual test dataset analysis

Innovation

• Uses LLM to analyze automotive tabular data
• Translates queries into Relational Algebra expressions
• Generates optimized Python code for analysis