Sola-Visibility-ISPM: Benchmarking Agentic AI for Identity Security Posture Management Visibility

2026-01-11

AI Summary

This work addresses the current lack of standardized methods for evaluating agents’ capabilities in performing Identity Security Posture Management (ISPM) visibility tasks within real-world enterprise environments. We propose the first reproducible benchmark for ISPM visibility, built upon a production-grade multi-cloud identity infrastructure spanning AWS, Okta, and Google Workspace, and comprising 77 natural language–driven evaluation tasks. We develop a tool-augmented agent with evidence-tracing capabilities that translates natural language queries into verifiable data exploration steps. Experimental results demonstrate that the agent achieves an overall expert accuracy of 0.84 and a strict success rate of 0.77, with AWS configuration hygiene tasks yielding the highest performance (accuracy: 0.94), thereby establishing a foundation for future research in advanced identity governance.

Technology Category

• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Multiagent Systems: Agent Communication
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Security and Privacy: Large-scale security measurements
• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Identity Security Posture Management (ISPM) is a core challenge for modern enterprises operating across cloud and SaaS environments. Answering basic ISPM visibility questions, such as understanding identity inventory and configuration hygiene, requires interpreting complex identity data, motivating growing interest in agentic AI systems. Despite this interest, there is currently no standardized way to evaluate how well such systems perform ISPM visibility tasks on real enterprise data. We introduce the Sola Visibility ISPM Benchmark, the first benchmark designed to evaluate agentic AI systems on foundational ISPM visibility tasks using a live, production-grade identity environment spanning AWS, Okta, and Google Workspace. The benchmark focuses on identity inventory and hygiene questions and is accompanied by the Sola AI Agent, a tool-using agent that translates natural-language queries into executable data exploration steps and produces verifiable, evidence-backed answers. Across 77 benchmark questions, the agent achieves strong overall performance, with an expert accuracy of 0.84 and a strict success rate of 0.77. Performance is highest on AWS hygiene tasks, where expert accuracy reaches 0.94, while results on Google Workspace and Okta hygiene tasks are more moderate, yet competitive. Overall, this work provides a practical and reproducible benchmark for evaluating agentic AI systems in identity security and establishes a foundation for future ISPM benchmarks covering more advanced identity analysis and governance tasks.

Problem

• Identity Security Posture Management
• Agentic AI
• Visibility
• Benchmarking
• Enterprise Identity

Innovation

• Agentic AI
• Identity Security Posture Management (ISPM)
• Visibility Benchmark
• Tool-using Agent
• Natural Language to Action