Local LLM Agents as Vulnerable Runtimes:A Source-Code Audit of the Agent Runtime Layer

2026-06-18

AI Summary

This work addresses the lack of code-level security auditing in locally deployed LLM agent runtimes, which introduces risks from previously unidentified vulnerabilities. We propose CLAWAUDIT, a static analysis framework that establishes the first vulnerability taxonomy tailored to this context—comprising five distinct categories—and introduces specialized detection rules to fill existing gaps. Guided by the STRIDE threat model, we developed 47 Semgrep YAML rules and 30 CodeQL queries, evaluated on our newly curated benchmark, OPENCLAWBENCH. Experimental results demonstrate that Semgrep and CodeQL achieve recall rates of 66.8% and 75.1%, respectively, on held-out test sets, with less than a 4-percentage-point performance gap between training and testing phases, thereby validating both the effectiveness and generalizability of our rule set.

Abstract

Local LLM agents such as OpenClaw and Nanobot run on end-user machines and act on host resources - the shell, filesystem, browser, stored credentials, and messaging applications - through natural-language goals. These agents have become privileged software runtimes that mediate between user intent, model outputs, and host-level actions. Existing research characterizes the landscape through prompt injection, malicious skills, marketplace risks, or black-box evaluation of agents. But the implementation layer that performs this mediation, the prompt builder, parser, tool dispatcher, skill loader, memory writer, network client, and permission gate, has remained an unexamined safety boundary. To our knowledge, no prior work has examined the agent's source tree to audit these components for implementation-level security weaknesses. We present CLAWAUDIT, a static auditing framework for measuring vulnerability exposure in local LLM agent runtimes. CLAWAUDIT derives a five-category vulnerability taxonomy from STRIDE and develops custom static-analysis rules that target agent-specific patterns absent from established rule sets for vulnerability analysis. We instantiate the taxonomy in two backends, 47 Semgrep YAML rules and 30 CodeQL queries, and evaluate on OPENCLAWBENCH, a benchmark of 446 source-code-level advisories from the OpenClaw repository and split temporally into 229 rule-derivation (train) and 217 held-out (test) advisories. On the held-out test, CLAWAUDIT raises Semgrep recall from 21.7% (Pro baseline) to 66.8%, and CodeQL recall from 13.8% (security-extended) to 75.1%. Train/test gaps remain within 4 percentage points for all four configurations, indicating that the rules generalize to vulnerabilities unseen during rule writing. A preliminary live-code audit shows that these recall-oriented rules require manual triage, motivating semantic filtering before production deployment.

Problem

• Local LLM agents
• Runtime security
• Source-code audit
• Vulnerability taxonomy
• Implementation-level weaknesses

Innovation

• Local LLM Agents
• Runtime Security
• Static Analysis
• Vulnerability Taxonomy
• CLAWAUDIT