RADIOv2.5: Improved Baselines for Agglomerative Vision Foundation Models

2024-12-10

AI Summary

This work addresses four key challenges in training aggregation-based vision foundation models: (1) difficulty in multi-resolution adaptation, (2) capability imbalance across multiple teacher models, (3) distillation artifacts, and (4) output token redundancy. Methodologically, we propose a novel multi-resolution collaborative training paradigm and a token-aware compression mechanism, integrated with mosaic augmentation, dynamically weighted teacher loss balancing, and multi-source knowledge distillation from CLIP, DINO, and SAM. Technically, our approach overcomes modality inconsistency and information density bottlenecks, enabling high-fidelity, efficient cross-scale knowledge transfer. We release four open-source model variants—B, L, H, and g—and demonstrate significant improvements over baselines on zero-shot transfer and fine-grained visual understanding tasks. All pre-trained weights and inference code are publicly available.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Multiagent Systems: Other Foundations of Multi Agent Systems

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Agglomerative models have recently emerged as a powerful approach to training vision foundation models, leveraging multi-teacher distillation from existing models such as CLIP, DINO, and SAM. This strategy enables the efficient creation of robust models, combining the strengths of individual teachers while significantly reducing computational and resource demands. In this paper, we thoroughly analyze state-of-the-art agglomerative models, identifying critical challenges including resolution mode shifts, teacher imbalance, idiosyncratic teacher artifacts, and an excessive number of output tokens. To address these issues, we propose several novel solutions: multi-resolution training, mosaic augmentation, and improved balancing of teacher loss functions. Specifically, in the context of Vision Language Models, we introduce a token compression technique to maintain high-resolution information within a fixed token count. We release our top-performing variants at multiple scales (-B, -L, -H, and -g), along with inference code and pretrained weights

Problem

• Address resolution mode shifts in agglomerative models
• Mitigate teacher imbalance and idiosyncratic artifacts
• Reduce excessive output tokens in Vision Language Models

Innovation

• Multi-resolution training
• Mosaic augmentation
• Token compression technique