T-REN: Learning Text-Aligned Region Tokens Improves Dense Vision-Language Alignment and Scalability

2026-04-20

AI Summary

Existing vision-language encoders face limitations in fine-grained semantic alignment and computational efficiency, hindering their performance on open-vocabulary segmentation and long-form video understanding. This work proposes T-REN, a lightweight region encoding network that aggregates image patches into text-aligned region-level tokens atop a frozen visual backbone, drastically reducing token count by 24× for images and 187× for videos. T-REN establishes the first efficient and compact text-aligned regional representation, substantially enhancing dense cross-modal alignment with minimal parameter overhead. Experimental results demonstrate significant improvements across multiple benchmarks: +5.9 mIoU on ADE20K, +18.4% Recall on COCO retrieval, +15.6% on Ego4D temporal localization, and +17.6% mIoU on VSPW parsing.

Technology Category

• Computer Vision: Multi-modal Vision
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Despite recent progress, vision-language encoders struggle with two core limitations: (1) weak alignment between language and dense vision features, which hurts tasks like open-vocabulary semantic segmentation; and (2) high token counts for fine-grained visual representations, which limits scalability to long videos. This work addresses both limitations. We propose T-REN (Text-aligned Region Encoder Network), an efficient encoder that maps visual data to a compact set of text-aligned region-level representations (or region tokens). T-REN achieves this through a lightweight network added on top of a frozen vision backbone, trained to pool patch-level representations within each semantic region into region tokens and align them with region-level text annotations. With only 3.7% additional parameters compared to the vision-language backbone, this design yields substantially stronger dense cross-modal understanding while reducing the token count by orders of magnitude. Specifically, T-REN delivers +5.9 mIoU on ADE20K open-vocabulary segmentation, +18.4% recall on COCO object-level text-image retrieval, +15.6% recall on Ego4D video object localization, and +17.6% mIoU on VSPW video scene parsing, all while reducing token counts by more than 24x for images and 187x for videos compared to the patch-based vision-language backbone. The code and model are available at https://github.com/savya08/T-REN.

Problem

• vision-language alignment
• dense representation
• token scalability
• open-vocabulary segmentation
• region-level representation

Innovation

• region tokens
• dense vision-language alignment
• text-aligned representation
• token reduction
• scalable vision-language modeling