White-Basilisk: A Hybrid Model for Code Vulnerability Detection

2025-07-11

AI Summary

To address the challenges of limited context length and high computational overhead in large language models (LLMs) for code vulnerability detection, this paper proposes a lightweight hybrid architecture (200M parameters) integrating the Mamba state-space model, linear self-attention, and a sparse Mixture-of-Experts (MoE) structure—enabling ultra-long code sequence modeling and end-to-end security analysis. Leveraging efficient training strategies, the model achieves state-of-the-art performance on real-world, imbalanced vulnerability datasets. It processes code snippets up to 10K tokens per inference pass, reduces GPU memory consumption by 62%, and accelerates inference by 3.1×. Results demonstrate that domain-specialized compact models—through architectural innovation—can significantly outperform general-purpose LLMs, striking a superior balance among accuracy, efficiency, and deployment cost.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Security and Privacy: Large-scale security measurements

Abstract

The proliferation of software vulnerabilities presents a significant challenge to cybersecurity, necessitating more effective detection methodologies. We introduce White-Basilisk, a novel approach to vulnerability detection that demonstrates superior performance while challenging prevailing assumptions in AI model scaling. Utilizing an innovative architecture that integrates Mamba layers, linear self-attention, and a Mixture of Experts framework, White-Basilisk achieves state-of-the-art results in vulnerability detection tasks with a parameter count of only 200M. The model's capacity to process sequences of unprecedented length enables comprehensive analysis of extensive codebases in a single pass, surpassing the context limitations of current Large Language Models (LLMs). White-Basilisk exhibits robust performance on imbalanced, real-world datasets, while maintaining computational efficiency that facilitates deployment across diverse organizational scales. This research not only establishes new benchmarks in code security but also provides empirical evidence that compact, efficiently designed models can outperform larger counterparts in specialized tasks, potentially redefining optimization strategies in AI development for domain-specific applications.

Problem

• Detects code vulnerabilities more effectively than current methods
• Challenges assumptions about AI model scaling for security tasks
• Processes long code sequences beyond current LLM limitations

Innovation

• Hybrid Mamba layers and linear self-attention
• Mixture of Experts for efficient scaling
• Processes long code sequences in single pass