Safeguarding LLM Agents from Misalignment through Provenance Analysis

2026-05-01

AI Summary

This work addresses the critical safety risks arising from intent misalignment when large language model (LLM) agents invoke external tools. To tackle this issue, the authors introduce ProvenanceGuard, the first framework to incorporate provenance analysis into LLM alignment. ProvenanceGuard employs a multi-stage defensive pipeline that, prior to tool execution, verifies whether the proposed tool call is supported by traceable evidence within the conversational context, thereby accurately identifying three distinct types of intent-tool mismatches. The approach establishes a structured and auditable detection mechanism, overcoming the inconsistency and opacity inherent in conventional LLM-as-a-judge methods. Experimental results demonstrate that ProvenanceGuard reduces misjudgment rates of mismatched trajectories from 44.3% to 2.1% on AgentSafetyBench and from 32.4% to 18.7% on WorkBench, while substantially minimizing interference with legitimate task execution.

Abstract

As LLM agents gain increasing access to powerful tools, ensuring that their actions are aligned with the user's intent becomes critical. When an agent's proposed tool invocation deviates from the user's intent -- a phenomenon called misalignment -- it may lead to harmful consequences that are difficult to undo. Existing runtime guardrails rely on an LLM-as-a-judge paradigm that lacks a systematic framework for reasoning about alignment, often producing judgments that are inconsistent or difficult to audit. Motivated by provenance analysis, we propose a provenance-based conceptual framework that formalizes misalignment detection as determining whether a proposed tool call is supported by traceable evidence in the agent's context. Building on this framework, we propose ProvenanceGuard, a multi-stage pipeline that analyzes the agent's action for three types of misalignment before the selected tool is executed and only allows the action to take place when it is considered aligned with the user's input query. We evaluated our proposed approach on two different benchmarks, Agent-SafetyBench and WorkBench, across 10 backbone LLMs. Compared to the LLM-as-a-judge baseline, ProvenanceGuard reduces error rate on misaligned traces from 42.9% to 1.8% on Agent-SafetyBench and from 32.1% to 17.3% on WorkBench, while reducing intervention burden on task-successful traces from 30.5% to 12.8% and introducing no statistically significant increase in unnecessary interventions on aligned traces. These results demonstrate that structured, provenance-based reasoning provides an effective and practical foundation for safeguarding LLM agents from misalignment.

Problem

• misalignment
• LLM agents
• provenance analysis
• tool misuse
• alignment safety

Innovation

• provenance analysis
• misalignment detection
• LLM agents
• runtime guardrails
• tool-call alignment