MINES: Explainable Anomaly Detection through Web API Invariant Inference

2025-12-07

AI Summary

Modern web applications (e.g., RESTful, SOAP, WebSocket APIs) are vulnerable to malicious attacks or misuse; their anomalous behavior logs closely resemble normal ones and often lack critical database context, compounded by pervasive logging noise—causing log-based learning methods to capture spurious correlations, resulting in poor generalization and interpretability. This paper proposes the first anomaly detection method that jointly infers *interpretable semantic invariants* at both the API interface and database schema levels: leveraging LLMs to parse table structures, uncover implicit constraints between API signatures and database schemas, and compile them into executable Python validation rules. High-confidence invariants are distilled from benign logs, enabling precise runtime anomaly detection. Evaluated on real-world systems—including TrainTicket and NiceFish—the approach achieves high recall and near-zero false positives, significantly outperforming LogRobust and LogFormer and establishing new state-of-the-art performance.

Technology Category

• Data Mining & Knowledge Management: Anomaly/Outlier Detection
• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Detecting the anomalies of web applications, important infrastructures for running modern companies and governments, is crucial for providing reliable web services. Many modern web applications operate on web APIs (e.g., RESTful, SOAP, and WebSockets), their exposure invites intended attacks or unintended illegal visits, causing abnormal system behaviors. However, such anomalies can share very similar logs with normal logs, missing crucial information (which could be in database) for log discrimination. Further, log instances can be also noisy, which can further mislead the state-of-the-art log learning solutions to learn spurious correlation, resulting superficial models and rules for anomaly detection. In this work, we propose MINES which infers explainable API invariants for anomaly detection from the schema level instead of detailed raw log instances, which can (1) significantly discriminate noise in logs to identify precise normalities and (2) detect abnormal behaviors beyond the instrumented logs. Technically, MINES (1) converts API signatures into table schema to enhance the original database shema; and (2) infers the potential database constraints on the enhanced database schema to capture the potential relationships between APIs and database tables. MINES uses LLM for extracting potential relationship based on two given table structures; and use normal log instances to reject and accept LLM-generated invariants. Finally, MINES translates the inferred constraints into invariants to generate Python code for verifying the runtime logs. We extensively evaluate MINES on web-tamper attacks on the benchmarks of TrainTicket, NiceFish, Gitea, Mastodon, and NextCloud against baselines such as LogRobust, LogFormer, and WebNorm. The results show that MINES achieves high recall for the anomalies while introducing almost zero false positives, indicating a new state-of-the-art.

Problem

• Detects anomalies in web APIs to ensure reliable services
• Infers explainable invariants from schema to reduce log noise
• Uses LLM and constraints for precise anomaly detection with low false positives

Innovation

• Uses LLM to infer API-database constraints from schema
• Converts API signatures into enhanced table schema
• Generates Python code for runtime log verification