Detecting Anomalies in Machine Learning Infrastructure via Hardware Telemetry

2025-10-29

AI Summary

Cloud-platform virtualization impedes operators’ observability into users’ machine learning (ML) workloads, hindering resource optimization and cost efficiency. To address this, we propose System-X—a novel framework that introduces the first unsupervised anomaly detection method leveraging hardware telemetry (e.g., performance counters) as its sole input, requiring no prior knowledge of workload characteristics. System-X enables operator-level system monitoring and optimization without instrumentation or model-specific assumptions. Evaluated across 30+ mainstream ML models on diverse hardware platforms, it demonstrates strong generalizability to emerging models and previously unseen deployment configurations. Experimental results show that System-X accurately identifies network and system configuration anomalies, significantly improving infrastructure efficiency—accelerating DeepSeek model training by 5.97%.

Technology Category

• Machine Learning: Hardware-aware ML
• Data Mining & Knowledge Management: Anomaly/Outlier Detection
• Humans and AI: Explainable AI (XAI) for Human Understanding

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Modern machine learning (ML) has grown into a tightly coupled, full-stack ecosystem that combines hardware, software, network, and applications. Many users rely on cloud providers for elastic, isolated, and cost-efficient resources. Unfortunately, these platforms as a service use virtualization, which means operators have little insight into the users' workloads. This hinders resource optimizations by the operator, which is essential to ensure cost efficiency and minimize execution time. In this paper, we argue that workload knowledge is unnecessary for system-level optimization. We propose System-X, which takes a emph{hardware-centric} approach, relying only on hardware signals -- fully accessible by operators. Using low-level signals collected from the system, System-X detects anomalies through an unsupervised learning pipeline. The pipeline is developed by analyzing over 30 popular ML models on various hardware platforms, ensuring adaptability to emerging workloads and unknown deployment patterns. Using System-X, we successfully identified both network and system configuration issues, accelerating the DeepSeek model by 5.97%.

Problem

• Detecting anomalies in ML infrastructure using hardware telemetry signals
• Addressing limited workload visibility due to cloud virtualization layers
• Optimizing resource efficiency without requiring application-level knowledge

Innovation

• Uses hardware telemetry for anomaly detection
• Implements unsupervised learning on hardware signals
• Analyzes diverse ML models for adaptability