EgoAdapt: A Multi-Scene Egocentric Adaptation Method for CVPR 2026 HD-EPIC VQA Challenge

2026-05-23

AI Summary

This work addresses the performance bottleneck in visual question answering (VQA) from first-person kitchen videos, where diverse question types and heterogeneous spatiotemporal semantics pose significant challenges. To tackle this, the authors propose EgoAdapt, a method that enables adaptive reasoning across seven heterogeneous question categories—such as recipe understanding, gaze estimation, and 3D perception—in the HD-EPIC benchmark. At inference time, EgoAdapt employs category-conditional routing, an answer scoring mechanism based on token-level likelihoods of alphabetic characters, and a consistency-based prediction fusion strategy leveraging option permutations and verification-style prompting. Experimental results demonstrate that EgoAdapt substantially outperforms existing baselines, achieving notable gains in overall accuracy on complex egocentric VQA tasks while maintaining a single unified model architecture.

Abstract

This technical report presents our solution, EgoAdapt (Egocentric Adaptation via Category, Calibration, and Consistency), to the CVPR 2026 HD-EPIC VQA challenge. HD-EPIC evaluates whether a vision-language model can reason over realistic first-person kitchen videos, where the evidence for an answer may be a short hand-object interaction, a long recipe trajectory, a spatial relation to a fixture, or a subtle gaze cue. The benchmark contains 26K multiple-choice questions across seven macro-categories: recipe, ingredient, nutrition, fine-grained action, 3D perception, object motion, and gaze. We observe that the main difficulty is not only model capacity, but also the mismatch between a single generic inference recipe and the heterogeneous temporal, spatial, and semantic structure of the benchmark. Our method, EgoAdapt, introduces three inference-time components: (1) category-conditioned routing with per-category prompts, frame budgets, and sampling rates; (2) calibrated option scoring that evaluates all candidate answers with letter-token likelihoods and generation agreement instead of relying only on direct generation; and (3) test-time consistency adaptation that aggregates predictions across option permutations and verification-style prompts for ambiguous cases. This design substantially improves over the available HD-EPIC baselines.

Problem

• egocentric vision
• visual question answering
• first-person video
• heterogeneous reasoning
• HD-EPIC

Innovation

• Egocentric VQA
• Category-Conditioned Routing
• Calibrated Option Scoring
• Test-Time Consistency Adaptation
• First-Person Video Reasoning