computer vision

This work addresses the challenge of building video-language understanding systems with human-like perceptual capabilities, enabling synergistic modeling of linguistic and dynamic visual temporal sequences. We systematically survey model architectures, training paradigms, and data construction methodologies in this domain, and introduce— for the first time—a unified, cross-perspective taxonomy that exposes core challenges including multimodal temporal alignment and dataset bias. Leveraging Transformer-based fusion, contrastive/generative pretraining, synthetic data augmentation, and benchmarks such as How2QA and Ego4D, we conduct a comprehensive, reproducible horizontal evaluation of state-of-the-art models under a standardized assessment protocol. Our key contributions are: (1) the first structured analytical framework for joint video-language modeling; (2) clear identification of critical research directions; and (3) a practical, deployable technology roadmap for embodied intelligence.

This work addresses two key challenges in few-shot visual detection (<1,000 images): the strong data dependency of CNNs and the low fine-tuning efficiency of multimodal large language models (MLLMs). We propose a specialized fine-tuning paradigm for text-annotated detection tasks, explicitly modeling the synergy between language guidance and visual localization. Our approach integrates prompt engineering, dynamic context reasoning, and lightweight end-to-end fine-tuning. Under extreme data scarcity, the method achieves a 36% mAP improvement over conventional CNN baselines—matching or even surpassing their fully supervised performance. To our knowledge, this is the first work to empirically demonstrate the strong generalization capability of MLLMs in domain-specific few-shot visual detection. It establishes a novel cross-modal few-shot learning paradigm grounded in task-aware textual supervision. The implementation is publicly available.

To address performance limitations in object detection and semantic segmentation under complex scenarios—including occlusion, small objects, and cross-domain generalization—this paper proposes a novel multimodal detection paradigm synergizing large language models (LLMs). Methodologically, it systematically integrates CNNs, YOLOv5/v8, and DETR architectures into an LLM-augmented inference framework, augmented by scalable data pipelines, model pruning, and quantization, and evaluated via a multi-dimensional metric system based on mAP and mIoU. Key contributions include: (1) bridging the gap between traditional feature engineering and end-to-end deep learning; (2) introducing a dynamic context enhancement mechanism tailored for challenging environments; and (3) achieving state-of-the-art accuracy-efficiency trade-offs on COCO and ADE20K. The fully open-sourced, reproducible framework significantly improves model generalizability and robustness across diverse real-world conditions.

This study systematically investigates the transferability of Vision Transformers (ViTs) and Convolutional Neural Networks (CNNs) as backbones for few-shot geometric estimation—specifically 2D rigid transformation estimation and fundamental matrix prediction. Motivated by the unclear generalization capabilities of pretrained models in low-data regimes (e.g., autonomous driving, robotics), we fine-tune diverse pretrained models—including ResNet, EfficientNet, CLIP-ResNet, CLIP-ViT, and DINO—within a unified framework. Results show that CNNs, benefiting from strong inductive biases, achieve stable performance and superior cross-domain generalization under extreme data scarcity (≤1k samples), whereas ViTs significantly outperform CNNs with larger training sets and demonstrate greater out-of-distribution robustness. Our key contribution is the first empirical revelation of a strong coupling between architectural choice and data scale in geometric vision tasks, leading to the practical principle of “backbone selection conditioned on data scale.” This work establishes a reproducible benchmark and design paradigm for resource-constrained geometric learning.

To address the training difficulty of ultra-deep convolutional neural networks (CNNs) caused by vanishing gradients, this paper proposes a residual learning framework. It introduces identity-mapping shortcut connections that allow input signals to bypass certain nonlinear transformations, thereby ensuring unimpeded forward signal propagation and backward gradient flow. This approach enables stable end-to-end training of CNNs with over 100 layers—e.g., ResNet-110—on CIFAR-10, achieving 89.9% accuracy, significantly surpassing comparably deep conventional CNNs (84.1%) while exhibiting faster convergence and greater training robustness. The core innovation lies in reformulating the learning objective from approximating a target mapping to learning a residual function, synergistically combined with batch normalization. This fundamentally alleviates optimization challenges inherent in very deep networks and establishes a foundational paradigm for deep model architecture design.

This paper addresses the challenge of unifying image understanding and generation in multimodal models by proposing the first diffusion Transformer architecture that directly generates CLIP image features. Methodologically: (1) it abandons the VAE latent space and instead performs diffusion-based generation in the semantically rich CLIP feature space; (2) it introduces a two-stage pretraining paradigm—“understanding-first, then generation”—to jointly optimize comprehension and synthesis capabilities; (3) it leverages GPT-4o to automatically construct BLIP3o-60k, a high-quality instruction-tuning dataset. Contributions include state-of-the-art performance across major understanding benchmarks (VQAv2, OK-VQA) and generation metrics (FID, CLIP-Score); 37% higher training efficiency; significantly improved generation fidelity; and full-stack open-sourcing of code, model weights, training scripts, and data.

Convolutional neural networks (CNNs) struggle to model global contextual dependencies in images. Method: This work systematically evaluates pure vision transformers (ViTs), hierarchical ViTs (e.g., Swin, CvT), and hybrid architectures across image classification (ImageNet), object detection (COCO), and medical image classification (ChestX-ray14), benchmarking all against unified CNN baselines and quantifying accuracy–efficiency trade-offs. Contribution/Results: It presents the first cross-task, cross-domain comparative evaluation of diverse ViT families and introduces a medical imaging–specific data augmentation strategy. Hierarchical ViTs consistently outperform CNNs: achieving +3.2% average AUC on ChestX-ray14 and 51.7% mAP on COCO. Results empirically validate the structural advantage of self-attention for modeling long-range dependencies, providing evidence-based guidance for deploying ViTs in both clinical and general-purpose vision applications.

This work addresses the limited accessibility of existing interpretability methods for Vision Transformers (ViTs), which often focus on isolated components or cater primarily to experts, thereby lacking intuitive tools for understanding end-to-end reasoning. To bridge this gap, the authors propose a web-based interactive visualization system that uniquely integrates animated demonstrations, patch-level attention heatmaps, and a visually adapted Logit Lens, collectively illustrating the full inference pipeline—from image tokenization to final classification. The system supports both guided learning and free exploration modes, substantially lowering the barrier to comprehension for non-expert users. User studies demonstrate that the tool is not only easy to use but also effectively enhances users’ understanding of ViT internal mechanisms.

To address the challenges of high-resolution image synthesis and multimodal semantic understanding, this paper introduces VLM-RF, a vision-enhanced large language model. Methodologically, it pioneers a noise-aware learning algorithm and integrates a linear-path rectified flow (RF) mechanism with a cross-modal bidirectional tokenization strategy, enabling unified spatiotemporal feature embedding and hybrid sequence modeling across text, images, and video. The contributions are threefold: (1) substantial improvement in generation quality—image resolution and perceptual sharpness increase by 25%; (2) 20% reduction in computational overhead; and (3) consistent superiority over state-of-the-art diffusion models in both synthesis fidelity and cross-modal alignment. By unifying generative modeling and language understanding within a single scalable architecture, VLM-RF establishes a novel paradigm for efficient, high-fidelity multimodal generation.

This study investigates the impact of fusion timing on the accuracy–latency trade-off in multimodal vision–language systems. We propose and systematically evaluate three fusion strategies—early, middle, and late—within a unified architecture combining BERT for language and lightweight visual backbones (MobileNetV2 or ViT) on the CMU MOSI dataset; inference latency is empirically measured on an NVIDIA Jetson Orin AGX edge platform. Results show that late fusion achieves the highest accuracy (12.3% lower MAE), while early fusion incurs the lowest latency (41.7% reduction on average), with fusion stage exhibiting a strong negative correlation between accuracy and latency. To our knowledge, this is the first work to quantitatively and systematically validate the critical influence of fusion location under a consistent experimental framework. Our findings provide reproducible architectural guidelines and empirical evidence for designing efficient multimodal models tailored to resource-constrained edge devices.

Existing video understanding research overlooks how structural characteristics of datasets—such as motion complexity, temporal span, hierarchical composition, and multimodal richness—guide the evolution of model architectures. Method: We propose a dataset-centric analytical framework that systematically interprets mainstream architectures—including two-stream networks, 3D CNNs, RNNs, Transformers, and multimodal foundation models—as responses to dataset-imposed inductive biases. Our approach integrates literature review with architecture–bias–task alignment analysis, unifying inductive bias theory and multimodal learning paradigms. Contribution/Results: We establish, for the first time, a unified “dataset → inductive bias → model design” framework, revealing the intrinsic logic underlying architectural evolution. The framework yields principled, generalizable design guidelines for video understanding models that balance scalability and task adaptability, advancing both theoretical understanding and practical model development.