Runtime Monitoring of Perception-Based Autonomous Systems via Embedding Temporal Logic

2026-05-12

AI Summary

Traditional runtime monitoring approaches struggle to effectively map high-dimensional continuous perceptual data into discrete logical propositions, leading to semantic mismatch, high computational overhead, and insufficient robustness. This work proposes Embedded Temporal Logic (ETL), which, for the first time, directly constructs temporal logic monitoring within a deep perceptual embedding space. ETL defines predicates based on distances between observation embeddings and reference embeddings, eliminating the need for explicit symbolic mapping. This formulation naturally supports high-level specifications such as visual similarity and semantic region avoidance, and integrates conformal prediction calibration to ensure safe and reliable predicate evaluation. Experiments demonstrate that ETL achieves strong alignment with ground-truth semantics across diverse environments, accurately monitors complex temporal compositions of behaviors, and significantly outperforms conventional methods relying on discrete abstractions.

Abstract

Runtime monitoring of autonomous systems traditionally relies on mapping continuous sensor observations to discrete logical propositions defined over low-dimensional state variables. This abstraction breaks down in perception-driven settings, where such mappings require additional learned modules that are often computationally expensive, brittle, and semantically misaligned. In this work, we propose Embedding Temporal Logic (ETL), a temporal logic that performs monitoring directly in learned embedding spaces. ETL defines predicates through distances between observed embeddings and target embeddings derived from reference observations. This formulation allows specifications to capture high-level perceptual concepts, such as similarity to visual goals or avoidance of semantic regions, that are difficult or impossible to express using traditional predicates. By composing these predicates with temporal operators, ETL naturally expresses temporally extended and sequential perceptual behaviors. We introduce ETL monitors for evaluating specifications over bounded embedding traces, along with a conformal calibration procedure that provides reliable and safety-oriented predicate evaluation. We evaluate our approach across multiple manipulation environments to show that ETL achieves strong empirical agreement with ground-truth semantics, including accurate monitoring of temporally composed behaviors.

Problem

• runtime monitoring
• perception-based autonomous systems
• temporal logic
• embedding spaces
• semantic alignment

Innovation

• Embedding Temporal Logic
• runtime monitoring
• perception-based autonomy
• conformal calibration
• learned embeddings