Structure-based Anomaly Detection and Clustering

2025-05-19

AI Summary

This paper addresses three core challenges: anomaly detection in structured and streaming data, structural clustering under noise, and open-set malware family identification. To tackle these, we propose a unified low-dimensional manifold modeling framework. For anomaly detection, we introduce Preference Isolation Forest (PIF) and its streaming variant, Sliding-PIF. For clustering multi-geometric structures corrupted by noise, we propose MultiLink—a model-aware hierarchical clustering algorithm leveraging Voronoi partitioning, locality-sensitive hashing (LSH), and manifold fitting. For evolving data streams, we design Online-iForest, enabling retraining-free online adaptation via sliding-window modeling. For open-set malware recognition, we integrate MaxLogit into open-set classification and deploy it in the industrial Cleafy system. Extensive experiments on synthetic and real-world benchmarks demonstrate state-of-the-art performance: MultiLink achieves superior robustness and efficiency; Online-iForest enables real-time detection; and MaxLogit is production-deployed.

Technology Category

• Data Mining & Knowledge Management: Anomaly/Outlier Detection
• Machine Learning: Clustering
• Humans and AI: Human-in-the-loop Machine Learning

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Web Mining and Content Analysis: Normalization, clustering, classification, and summarization of Web text

Abstract

Anomaly detection is a fundamental problem in domains such as healthcare, manufacturing, and cybersecurity. This thesis proposes new unsupervised methods for anomaly detection in both structured and streaming data settings. In the first part, we focus on structure-based anomaly detection, where normal data follows low-dimensional manifolds while anomalies deviate from them. We introduce Preference Isolation Forest (PIF), which embeds data into a high-dimensional preference space via manifold fitting, and isolates outliers using two variants: Voronoi-iForest, based on geometric distances, and RuzHash-iForest, leveraging Locality Sensitive Hashing for scalability. We also propose Sliding-PIF, which captures local manifold information for streaming scenarios. Our methods outperform existing techniques on synthetic and real datasets. We extend this to structure-based clustering with MultiLink, a novel method for recovering multiple geometric model families in noisy data. MultiLink merges clusters via a model-aware linkage strategy, enabling robust multi-class structure recovery. It offers key advantages over existing approaches, such as speed, reduced sensitivity to thresholds, and improved robustness to poor initial sampling. The second part of the thesis addresses online anomaly detection in evolving data streams. We propose Online Isolation Forest (Online-iForest), which uses adaptive, multi-resolution histograms and dynamically updates tree structures to track changes over time. It avoids retraining while achieving accuracy comparable to offline models, with superior efficiency for real-time applications. Finally, we tackle anomaly detection in cybersecurity via open-set recognition for malware classification. We enhance a Gradient Boosting classifier with MaxLogit to detect unseen malware families, a method now integrated into Cleafy's production system.

Problem

• Proposes unsupervised anomaly detection in structured and streaming data
• Introduces structure-based clustering for noisy multi-model data recovery
• Develops online anomaly detection for evolving data streams efficiently

Innovation

• Preference Isolation Forest for manifold-based anomaly detection
• MultiLink clustering for multiple geometric model recovery
• Online Isolation Forest for evolving data streams