MuViT: Multi-Resolution Vision Transformers for Learning Across Scales in Microscopy

2026-02-27

AI Summary

Microscopic images exhibit multi-scale structures, yet existing vision models typically process only a single resolution, hindering effective cross-scale information integration. To address this, this work proposes MuViT—a Transformer-based multi-resolution architecture that, for the first time in microscopic image analysis, introduces explicit world-coordinate modeling. By embedding image patches from different resolutions into a unified coordinate system and extending rotational position encoding, MuViT enables fusion of cross-scale context and fine details within a single encoder. The framework supports multi-resolution masked autoencoding (MAE) pretraining to learn scale-consistent representations. Experiments demonstrate that MuViT significantly outperforms strong ViT and CNN baselines across synthetic data, kidney histopathology, and high-resolution mouse brain microscopy, consistently enhancing downstream task performance.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multi-instance/Multi-view Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Modern microscopy routinely produces gigapixel images that contain structures across multiple spatial scales, from fine cellular morphology to broader tissue organization. Many analysis tasks require combining these scales, yet most vision models operate at a single resolution or derive multi-scale features from one view, limiting their ability to exploit the inherently multi-resolution nature of microscopy data. We introduce MuViT, a transformer architecture built to fuse true multi-resolution observations from the same underlying image. MuViT embeds all patches into a shared world-coordinate system and extends rotary positional embeddings to these coordinates, enabling attention to integrate wide-field context with high-resolution detail within a single encoder. Across synthetic benchmarks, kidney histopathology, and high-resolution mouse-brain microscopy, MuViT delivers consistent improvements over strong ViT and CNN baselines. Multi-resolution MAE pretraining further produces scale-consistent representations that enhance downstream tasks. These results demonstrate that explicit world-coordinate modelling provides a simple yet powerful mechanism for leveraging multi-resolution information in large-scale microscopy analysis.

Problem

• multi-resolution
• microscopy
• vision transformers
• spatial scales
• image analysis

Innovation

• Multi-resolution Vision Transformer
• World-coordinate Embedding
• Rotary Positional Embeddings
• Scale-consistent Representation
• Multi-resolution MAE Pretraining