Two-Stream Interactive Joint Learning of Scene Parsing and Geometric Vision Tasks

2026-02-14

AI Summary

This work addresses the challenge of efficiently jointly learning scene parsing and geometric vision tasks without relying on costly human-annotated correspondences. Inspired by the human two-pathway visual system, the authors propose TwInS, a dual-stream interactive learning framework that enables selective fusion of multi-level heterogeneous features through cross-task adapters. They further introduce the first semi-supervised training strategy that operates without ground-truth alignment between tasks, leveraging multi-view geometric constraints to drive model self-evolution. Extensive experiments on three public benchmarks demonstrate that the proposed method significantly outperforms current state-of-the-art approaches, validating the effectiveness and superiority of both its architectural design and training mechanism.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multi-instance/Multi-view Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Inspired by the human visual system, which operates on two parallel yet interactive streams for contextual and spatial understanding, this article presents Two Interactive Streams (TwInS), a novel bio-inspired joint learning framework capable of simultaneously performing scene parsing and geometric vision tasks. TwInS adopts a unified, general-purpose architecture in which multi-level contextual features from the scene parsing stream are infused into the geometric vision stream to guide its iterative refinement. In the reverse direction, decoded geometric features are projected into the contextual feature space for selective heterogeneous feature fusion via a novel cross-task adapter, which leverages rich cross-view geometric cues to enhance scene parsing. To eliminate the dependence on costly human-annotated correspondence ground truth, TwInS is further equipped with a tailored semi-supervised training strategy, which unleashes the potential of large-scale multi-view data and enables continuous self-evolution without requiring ground-truth correspondences. Extensive experiments conducted on three public datasets validate the effectiveness of TwInS's core components and demonstrate its superior performance over existing state-of-the-art approaches. The source code will be made publicly available upon publication.

Problem

• scene parsing
• geometric vision
• joint learning
• semi-supervised learning
• multi-view data

Innovation

• Two-Stream Architecture
• Interactive Joint Learning
• Cross-Task Adapter
• Semi-Supervised Training
• Geometric Vision