ReasonCLIP-58M: Visually Grounded Commonsense Reasoning Supervision for CLIP

2026-06-25

AI Summary

This work addresses the limitations of CLIP in visual commonsense and compositional reasoning due to its pretraining paradigm, which hinders advanced multimodal reasoning in downstream tasks. To overcome this, the authors propose ReasonCLIP-58M—a method that enhances CLIP without altering its architecture—through a two-stage continual pretraining strategy incorporating structured commonsense reasoning supervision. The approach leverages two newly curated datasets, ReasonLite-42M and ReasonPro-16M, and is evaluated on a dedicated benchmark, RCLIP-Bench. Experimental results demonstrate that ReasonCLIP-58M substantially improves zero-shot commonsense and compositional reasoning capabilities. Notably, it provides plug-and-play performance gains both in retrieval tasks and when employed as the vision encoder in state-of-the-art multimodal large language models such as LLaVA-NeXT.

Abstract

CLIP and its variants are widely adopted visual backbones in multimodal systems, but their pretraining remains dominated by descriptive image-text alignment. As downstream applications increasingly demand visually grounded commonsense inference and compositional reasoning, it remains unclear whether CLIP-style encoders can support such reasoning without architectural changes. To address this, we present ReasonCLIP-58M, a continual pretraining framework that integrates large-scale reasoning supervision into CLIP-style models through our two-stage strategy, which progressively integrates reasoning signals while preserving descriptive alignment, followed by category-structured reasoning supervision. To support this framework, we construct two complementary datasets and a benchmark: ReasonLite-42M, with open-form, visually verifiable reasoning captions; ReasonPro-16M, with category-specific reasoning supervision; and RCLIP-Bench for diagnostic evaluation of visually grounded reasoning. We train a family of ReasonCLIP that improves visually grounded commonsense and compositional reasoning while also enhancing zero-shot retrieval performance. As a drop-in visual encoder for multimodal large language models such as LLaVA-NeXT, ReasonCLIP delivers consistent gains without additional inference cost, demonstrating that structured reasoning supervision enhances the expressive capacity of CLIP-style visual representations. All datasets, models, and training code are available at https://github.com/RISys-Lab/ReasonCLIP.

Problem

• visually grounded reasoning
• commonsense reasoning
• compositional reasoning
• CLIP
• multimodal representation

Innovation

• visually grounded reasoning
• continual pretraining
• compositional reasoning
• CLIP-style models
• reasoning supervision