VL2Spike: Spike-driven Distillation from VLMs for Low-Power Visual Perception in Embodied AI

2026-06-14

AI Summary

This work addresses the performance limitations of spiking neural networks in visual perception tasks, which stem from their spike-based mechanisms and optimization challenges, thereby hindering the realization of their inherent energy efficiency. To overcome this, the authors propose VL2Spike, a framework that significantly enhances the representational capacity of Spikformer through spike-driven knowledge distillation from a vision-language model (VLM). The approach introduces two key innovations: spatio-temporal visual spike (SVS) distillation and spike prototype-guided language (SPL) distillation, enabling effective alignment between multimodal knowledge and spiking dynamics. Experiments demonstrate that VL2Spike achieves an average accuracy improvement of 6.81% on three static image datasets while consuming only 15.7% of the energy, and further yields a 6.63% performance gain in robotic visual place recognition, exhibiting strong generalization and superior energy efficiency.

Abstract

Spiking neural networks (SNNs) are brain-inspired, event-driven models that compute with sparse spikes, which enables highly efficient visual perception in resource-constrained embodied AI models. The emergence of Spiking-Transformer models with spike self-attention has substantially improved the learning capacity of pure SNNs. Although SNNs are energy efficient, their performance is still limited by the spike-based architecture and optimization challenges, as standard gradient descent rules cannot be directly applied. Recently, vision-language models (VLMs) have shown rich multi-modal knowledge representation capabilities for visual perception. Thus, it is promising to leverage VLMs for better Spikformer training. To this end, we present VL2Spike, a novel spike-based knowledge distillation (KD) framework that bridges multi-modal knowledge from VLMs with compact Spikformer models. This design enhances the learning capacity of Spikformer models while preserving their energy-efficiency merits, thereby offering a practical pathway toward low-power robotic perception. Our VL2Spike brings two key technical contributions. To align with spiking dynamics, we first propose spatial-temporal visual spike (SVS) distillation, which achieves (1) shared manifold alignment between VLM image features and spike tokens, and (2) warm-started temporal consistency on membrane potentials and spike rates. We then design a novel spike prototype-guided linguistic (SPL) distillation strategy that aligns Spikformer's class prototypes and logits with promptable VLM text embeddings. Extensive experiments show that VL2Spike achieves 6.81% gain across three static datasets with only 15.7% energy consumption. It also exhibits strong generalization capacity on robotic visual place recognition (VPR) with a gain of 6.63%, highlighting its potential for low-power perception in embodied AI.

Problem

• Spiking Neural Networks
• Vision-Language Models
• Knowledge Distillation
• Low-Power Perception
• Embodied AI

Innovation

• Spiking Neural Networks
• Vision-Language Models
• Knowledge Distillation
• Spikformer
• Low-Power Perception