Observability and Fault Injection for LLM-Based Multi-Agent Systems in Software Engineering

2026-08-25

AI Summary

为解决基于大语言模型的多智能体系统难以检查、调试和评估的问题，提出了一种结合OpenTelemetry分布式追踪与故障注入的轻量级工具llmmas-otel。

Abstract

Large Language Model-based multi-agent systems are increasingly explored for software engineering tasks, but they remain difficult to inspect, debug, and evaluate under controlled failures. We present llmmas-otel, a lightweight and framework-agnostic tool that combines OpenTelemetry-based distributed tracing with fault injection for LLM-based multi-agent systems in software engineering workflows. The tool instruments agent executions with trace-aligned telemetry across workflow phases, agent steps, inter-agent communication, tool calls, and LLM invocations, and supports targeted fault injection at selected interaction points. This makes it possible to compare baseline and faulty executions in a reproducible way and inspect the effects through aligned traces and run artifacts. We describe the motivation, architecture, implementation, current capabilities, and initial validation of the tool on a minimal demo workflow and a real LLM-based multi-agent system for software development.

Problem

• Large Language Model
• multi-agent systems
• software engineering
• observability
• fault injection

Innovation

• Distributed Tracing
• Fault Injection
• LLM-based Multi-Agent Systems
• OpenTelemetry