ESsEN: Training Compact Discriminative Vision-Language Transformers in a Low-Resource Setting

2026-04-20

AI Summary

This work addresses the challenges of high computational cost and substantial data requirements that hinder the deployment of vision-language models in low-resource settings, such as edge devices. The authors propose ESsEN, a lightweight discriminative dual-tower vision-language model that integrates convolutional networks with Transformers to enhance parameter efficiency. ESsEN incorporates a flexible and tunable cross-modal fusion module and employs an end-to-end training strategy tailored for low-resource scenarios. With only a small number of parameters and limited training data, ESsEN achieves performance comparable to that of large-scale models across multiple tasks, substantially lowering the resource barrier for effective vision-language modeling.

Technology Category

• Computer Vision: Language and Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Vision-language modeling is rapidly increasing in popularity with an ever expanding list of available models. In most cases, these vision-language models have parameters in the tens of billions, which is necessary for some needs, but in many cases smaller models are necessary (e.g., on edge devices or independent robotic platforms). Unfortunately, there is little research in producing light-weight models or in training them with small datasets. Inspired by the language learning progression and data sparsity in child development, in this paper, we address both of these goals in a systematic fashion. We show that two-tower encoder models are superior to one-tower encoders in low-resource settings for discriminative English tasks. We show also that incorporating traditional convolutional networks into the two-tower transformer architecture can help produce parameter efficient vision-language models. Finally, we show that the cross-modal fusion module of two-tower encoders can vary significantly in shape and size while producing the same results. In addition, we present ESsEN, a compact vision-language model that can be trained end-to-end with relatively few resources that performs as well on several tasks with only a fraction of the parameters compared to other models. The experimental results and the tools we present here make vision-language modeling more accessible to a wider variety of researchers.

Problem

• vision-language modeling
• low-resource setting
• compact models
• small datasets
• parameter efficiency

Innovation

• vision-language modeling
• low-resource training
• two-tower encoder
• parameter-efficient architecture
• compact model