SPENCER: Self-Adaptive Model Distillation for Efficient Code Retrieval

2025-08-01

AI Summary

In natural language-to-code retrieval, dual encoders suffer from limited accuracy due to insufficient fine-grained interaction, while cross-encoders incur high inference latency. To address this trade-off, we propose SPENCER: a collaborative dual-encoder coarse retrieval and cross-encoder fine-reranking framework that balances efficiency and accuracy. We introduce adaptive model distillation to reduce dual-encoder inference time by 70% while preserving over 98% of its original performance. Additionally, we design a pedagogical assistant selection strategy that dynamically selects the optimal teacher model based on pre-trained model characteristics, thereby synergistically optimizing knowledge transfer efficiency and effectiveness. Extensive experiments demonstrate that SPENCER significantly outperforms pure dual-encoder baselines across multiple benchmarks, achieving both high retrieval accuracy and low latency.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Code retrieval aims to provide users with desired code snippets based on users' natural language queries. With the development of deep learning technologies, adopting pre-trained models for this task has become mainstream. Considering the retrieval efficiency, most of the previous approaches adopt a dual-encoder for this task, which encodes the description and code snippet into representation vectors, respectively. However, the model structure of the dual-encoder tends to limit the model's performance, since it lacks the interaction between the code snippet and description at the bottom layer of the model during training. To improve the model's effectiveness while preserving its efficiency, we propose a framework, which adopts Self-AdaPtive Model Distillation for Efficient CodE Retrieval, named SPENCER. SPENCER first adopts the dual-encoder to narrow the search space and then adopts the cross-encoder to improve accuracy. To improve the efficiency of SPENCER, we propose a novel model distillation technique, which can greatly reduce the inference time of the dual-encoder while maintaining the overall performance. We also propose a teaching assistant selection strategy for our model distillation, which can adaptively select the suitable teaching assistant models for different pre-trained models during the model distillation to ensure the model performance. Extensive experiments demonstrate that the combination of dual-encoder and cross-encoder improves overall performance compared to solely dual-encoder-based models for code retrieval. Besides, our model distillation technique retains over 98% of the overall performance while reducing the inference time of the dual-encoder by 70%.

Problem

• Improves code retrieval accuracy via dual-encoder and cross-encoder interaction
• Reduces inference time while maintaining performance via model distillation
• Adaptively selects teaching assistant models for efficient distillation

Innovation

• Combines dual-encoder and cross-encoder for retrieval
• Uses self-adaptive model distillation for efficiency
• Adaptively selects teaching assistant models