Malicious ML Model Detection by Learning Dynamic Behaviors

2026-04-21

AI Summary

This work addresses the limitations of existing approaches for detecting malicious behavior in pre-trained models, which often rely on static analysis or handcrafted rules and consequently suffer from high false-negative or false-positive rates. To overcome these shortcomings, the authors propose a novel detection framework based on dynamic behavioral analysis. By monitoring the runtime behavior of benign models and integrating One-Class Support Vector Machines (OCSVM) with machine learning classifiers, the framework constructs a robust anomaly detector. This study is the first to combine dynamic analysis with OCSVM for this purpose, effectively mitigating the constraints inherent in static methods. Experimental evaluation on over 25,000 models demonstrates that the proposed approach achieves up to a 44% improvement in F1 score over the current state-of-the-art, substantially enhancing both accuracy and robustness in detecting malicious models.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Computer Vision: Adversarial Attacks & Robustness
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Pre-trained machine learning models (PTMs) are commonly provided via Model Hubs (e.g., Hugging Face) in standard formats like Pickles to facilitate accessibility and reuse. However, this ML supply chain setting is susceptible to malicious attacks that are capable of executing arbitrary code on trusted user environments, e.g., during model loading. To detect malicious PTMs, state-of-the-art detectors (e.g., PickleScan) rely on rules, heuristics, or static analysis, but ignore runtime model behaviors. Consequently, they either miss malicious models due to under-approximation (blacklisting) or miscategorize benign models due to over-approximation (static analysis or whitelisting). To address this challenge, we propose a novel technique (DynaHug) which detects malicious PTMs by learning the behavior of benign PTMs using dynamic analysis and machine learning (ML). DynaHug trains an ML classifier (one-class SVM (OCSVM)) on the runtime behaviours of task-specific benign models. We evaluate DynaHug using over 25,000 benign and malicious PTMs from different sources including Hugging Face and MalHug. We also compare DynaHug to several state-of-the-art detectors including static, dynamic and LLM-based detectors. Results show that DynaHug is up to 44% more effective than existing baselines in terms of F1-score. Our ablation study demonstrates that our design decisions (dynamic analysis, OCSVM, clustering) contribute positively to DynaHug's effectiveness.

Problem

• Malicious ML Model Detection
• Dynamic Behavior
• Pre-trained Models
• Model Hub Security
• Runtime Analysis

Innovation

• dynamic analysis
• one-class SVM
• malicious model detection
• pre-trained models
• behavior learning