PromAssistant: Leveraging Large Language Models for Text-to-PromQL

2025-03-05

AI Summary

Manually authoring PromQL queries for Prometheus monitoring poses high barriers for engineers, requiring both domain expertise and programming proficiency. Method: This work formally defines the text-to-PromQL task—automatically translating natural language questions into executable PromQL queries—and introduces the first collaborative reasoning framework integrating a domain-specific knowledge graph with large language models (LLMs) to support context-aware domain-specific language (DSL) generation. The framework incorporates prompt engineering, semantic parsing, and query validation to ensure correctness and robustness. Contribution/Results: Evaluated on a novel, manually curated benchmark of 280 real-world questions, the approach significantly outperforms existing baselines. It demonstrates the feasibility and effectiveness of natural language–driven metric analysis in observability, establishing a new paradigm for intelligent, accessible monitoring systems.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Data Mining & Knowledge Management: Intelligent Query Processing
• Knowledge Representation and Reasoning: Qualitative Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

With the increasing complexity of modern online service systems, understanding the state and behavior of the systems is essential for ensuring their reliability and stability. Therefore, metric monitoring systems are widely used and become an important infrastructure in online service systems. Engineers usually interact with metrics data by manually writing domain-specific language (DSL) queries to achieve various analysis objectives. However, writing these queries can be challenging and time-consuming, as it requires engineers to have high programming skills and understand the context of the system. In this paper, we focus on PromQL, which is the metric query DSL provided by the widely used metric monitoring system Prometheus. We aim to simplify metrics querying by enabling engineers to interact with metrics data in Prometheus through natural language, and we call this task text-to-PromQL. Building upon the insight, this paper proposes PromAssistant, a Large Language Model-based text-to-PromQL framework. PromAssistant first uses a knowledge graph to describe the complex context of an online service system. Then, through the synergistic reasoning of LLMs and the knowledge graph, PromAssistant transforms engineers' natural language questions into PromQL queries. To evaluate PromAssistant, we manually construct the first text-to-PromQL benchmark dataset which contains 280 metric query questions. The experiment results show that PromAssistant is effective in text-to-PromQL and outperforms baseline approaches. To the best of our knowledge, this paper is the first study of text-to-PromQL, and PromAssistant pioneered the DSL generation framework for metric querying and analysis.

Problem

• Simplify PromQL querying using natural language input.
• Enable engineers to interact with Prometheus metrics without high programming skills.
• Develop a framework for converting natural language questions into PromQL queries.

Innovation

• Leverages Large Language Models for text-to-PromQL
• Uses knowledge graph for system context understanding
• Synergistic reasoning between LLMs and knowledge graph