Bridging Vision and Language Concepts through Optimal Transport Semantic Flow

2026-06-25

AI Summary

Existing vision-language concept bottleneck models rely on static alignment or global similarity, limiting their ability to achieve fine-grained concept localization and capture the true semantic geometric structure. This work proposes OTF-CBM, the first model to integrate unbalanced optimal transport flows into concept bottleneck modeling. By learning semantic costs through inverse optimal transport and introducing a velocity-driven concept activation mechanism, OTF-CBM dynamically aligns image patches with textual concepts via semantic transport—without requiring numerical integration of ordinary differential equations. The method achieves state-of-the-art performance in both classification accuracy and concept fidelity, offering a novel geometric and dynamical perspective for interpretable cross-modal reasoning.

Abstract

Concept Bottleneck Models (CBMs) promise transparent reasoning by predicting through human-interpretable concepts, yet their effectiveness fundamentally depends on how well visual and textual representations are aligned or matched. Existing vision-language CBMs often rely on pre-aligned encoders or global cosine similarity, which obscures fine-grained concept localization and fails to reflect true semantic geometry. In this work, we rethink concept alignment as a dynamic cross-modal transport process instead of static projection and propose the Optimal Transport Flow Concept Bottleneck Model (OTF-CBM). It first learns a data-driven semantic cost via Inverse Optimal Transport to measure cross-modal distances, and then performs unbalanced optimal-transport-based flow matching to model semantic transitions between visual patches and textual concepts. With velocity-based concept activation, OTF-CBM captures interpretable geometric relations without ODE integration. Experiments further show that OTF-CBM achieves superior classification accuracy and concept faithfulness, offering a new geometric and dynamical perspective for interpretable cross-modal reasoning.

Problem

• Concept Bottleneck Models
• Vision-Language Alignment
• Semantic Geometry
• Fine-grained Concept Localization
• Cross-modal Reasoning

Innovation

• Optimal Transport
• Concept Bottleneck Models
• Cross-modal Alignment
• Semantic Flow
• Interpretable AI