CogniRoute: Learning to Route Social Evidence in Omni-Modal Models

2026-06-18

AI Summary

This work addresses the challenge that existing multimodal models struggle to effectively capture critical cross-modal cues—such as gestures, prosody, or audio-visual inconsistencies—in social video question answering. To this end, the authors propose CogniRoute, a cognitive schema-guided mixture-of-experts framework that decomposes training samples according to cross-modal relationships, reasoning types, and temporal scopes, and jointly optimizes expert routing and answer generation via routing-aware reinforcement learning. Key contributions include OmniSocialBench, the first fine-grained annotated benchmark for social video QA, a schema-driven routing mechanism, and a reward function integrating answer correctness, modal consistency, and temporal grounding. Experiments show that CogniRoute achieves 59.38% average accuracy on OmniSocialBench, outperforming the strongest closed-source and open-source baselines by 15.33 and 26.77 percentage points, respectively, with particularly strong performance on audio-visual coordination and conflict reasoning tasks.

Abstract

Omni-modal models can ingest video, audio, and text, but unified access to multiple modalities does not guarantee that a model uses the right evidence. This gap is especially pronounced in social video question answering, where the answer may hinge on a gesture, vocal tone, temporal cue, or mismatch between what is said and what is visually expressed. We introduce CogniRoute, a schema-guided Mixture-of-Experts framework for social omni reasoning. CogniRoute uses a training-only cognitive schema that factorizes each example by cross-modal relation, reasoning demand, and temporal scope, and aligns global routing signatures with this structure during supervised fine-tuning. We further introduce route-aware reinforcement learning, which jointly optimizes token generation and expert allocation using rewards for answer correctness, modality-consistent reasoning, and cognitive temporal grounding. To support training and evaluation, we construct OmniSocialBench, a diagnostic social video QA resource with 118K structured training examples, grounded reasoning traces, schema labels, temporal evidence spans, and a manually verified evaluation split. CogniRoute achieves 59.38\% average accuracy on OmniSocialBench, improving over the strongest proprietary baseline by 15.33 percentage points and the strongest open-source omni baseline by 26.77 points, with the largest gains on questions requiring audio-visual coordination, conflict resolution, and temporally grounded social inference.

Problem

• omni-modal models
• social video question answering
• cross-modal evidence
• temporal reasoning
• multimodal alignment

Innovation

• Mixture-of-Experts
• cognitive schema
• route-aware reinforcement learning
• omni-modal reasoning
• social video QA