t-gems: text-guided exit modules for decreasing clip image encoder

2026-05-17

AI Summary

This work addresses the high computational cost of multimodal models, which stems from their reliance on large image encoders and unified inference pathways, hindering efficient deployment. The authors propose Text-Guided Early-exit Modules (T-GEMs), a novel approach that leverages textual semantics to dynamically inform early-exit decisions in the image encoder—a strategy introduced for the first time in this context. By incorporating a rate-based regularization mechanism, T-GEMs significantly reduce the average computational load of the image encoder while preserving strong cross-modal understanding and retrieval performance. The method achieves a controllable trade-off between computational efficiency and model accuracy by analyzing semantic distributions in intermediate layers of vision-language models such as CLIP, thereby substantially improving inference efficiency.

Abstract

Multimodal deep neural networks enhance deep comprehension by integrating diverse data modalities. Data from different modalities are typically projected into a shared latent space for similarity computation, but this process is resource intensive due to large image encoders and equal processing of test data during prediction. Early exit methods reduce computational load by utilizing intermediate layers, saving time and memory. However, developing such methods is challenging for multimodal data like image-text pairs. This study investigates the semantic content distributions present in intermediate layers of encoders such as CLIP, which can be derived from textual descriptions. We introduce Text-Guided Exit Modules (T-GEMs) and a rate-based regularizer to control encoder usage costs while maintaining cross-modal understanding performance.

Problem

• multimodal neural networks
• early exit
• CLIP
• computational efficiency
• image-text pairs

Innovation

• early exit
• multimodal learning
• CLIP
• text-guided
• computational efficiency