AutonomyLens: A Self-Evolving Simulation-Based Testing Loop for Autonomous Systems

2026-04-13

AI Summary

Current verification workflows for autonomous systems suffer from a lack of coordination among scenario design, simulation execution, and telemetry analysis, leading to poor traceability between requirements, tests, and evidence, which undermines reproducibility and debugging efficiency. This work proposes a unified verification framework powered by large language models (LLMs) that bridges this gap through task-level structured scenario representations. The framework automatically translates high-level verification intents into temporally evolving scenarios, enabling automated simulation execution and context-aligned telemetry analysis. Furthermore, it incorporates a counterfactual scenario generation mechanism driven by failure cases to establish a closed-loop, self-evolving testing process. The approach substantially enhances traceability, reproducibility, and scalability of verification, accelerates test iteration cycles, and deepens insight into system behavior.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Multiagent Systems: Agent-Based Simulation and Emergent Behavior
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Software engineering practices for validating autonomous cyber-physical systems (e.g., Uncrewed Aerial Vehicles) remain fragmented across scenario design, simulation execution, and telemetry analysis, limiting traceability between requirements, tests, and evidence. This fragmentation reduces reproducibility, slows debugging and iteration, and hinders systematic assurance under complex and evolving environmental conditions. We present AutonomyLens, an LLM-driven framework that integrates scenario specification, simulation execution, and telemetry analysis into a unified validation workflow. AutonomyLens enables developers to translate high-level validation intent into executable, temporally evolving scenarios, automatically run simulations, and perform context-aware analysis of resulting system behavior. The framework introduces (i) a structured representation for mission-level scenarios, (ii) an automated execution pipeline, (iii) analysis mechanisms that align telemetry with scenario context to produce actionable insights, and (iv) counterfactual scenario generation that closes the loop by refining and synthesizing new test cases from observed failures. We describe the early-stage design of AutonomyLens, discuss key challenges in building integrated validation workflows for autonomous systems, and outline how such an approach can improve traceability, reproducibility, and scalability in autonomy validation.

Problem

• autonomous systems
• validation
• simulation
• traceability
• reproducibility

Innovation

• self-evolving testing
• LLM-driven validation
• scenario synthesis
• telemetry analysis
• autonomous systems