Ego4OOD: Rethinking Egocentric Video Domain Generalization via Covariate Shift Scoring

2026-01-21

AI Summary

This work addresses the challenge of generalizing egocentric video action recognition under domain shift, where large intra-class spatiotemporal variation, long-tailed feature distributions, and strong coupling between actions and environments hinder performance. To this end, we propose Ego4OOD—the first domain generalization benchmark derived from Ego4D—that explicitly targets covariate shift while mitigating concept shift through semantically consistent action partitioning. We further introduce a clustering-driven covariate shift score to quantify domain difficulty. Methodologically, we adopt a one-vs-all binary classification strategy, decomposing the multi-class task into independent binary subtasks, each solved by a lightweight two-layer fully connected network. Experiments demonstrate that our framework achieves state-of-the-art performance on both Ego4OOD and Argo1M with fewer parameters and without leveraging multimodal inputs, and for the first time empirically reveals a significant negative correlation between the degree of covariate shift and recognition accuracy.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Computer Vision: Video Understanding & Activity Analysis
• Application Domains: Mobility, Driving & Flight

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Vertical and domain-specific search

Abstract

Egocentric video action recognition under domain shifts remains challenging due to large intra-class spatio-temporal variability, long-tailed feature distributions, and strong correlations between actions and environments. Existing benchmarks for egocentric domain generalization often conflate covariate shifts with concept shifts, making it difficult to reliably evaluate a model's ability to generalize across input distributions. To address this limitation, we introduce Ego4OOD, a domain generalization benchmark derived from Ego4D that emphasizes measurable covariate diversity while reducing concept shift through semantically coherent, moment-level action categories. Ego4OOD spans eight geographically distinct domains and is accompanied by a clustering-based covariate shift metric that provides a quantitative proxy for domain difficulty. We further leverage a one-vs-all binary training objective that decomposes multi-class action recognition into independent binary classification tasks. This formulation is particularly well-suited for covariate shift by reducing interference between visually similar classes under feature distribution shift. Using this formulation, we show that a lightweight two-layer fully connected network achieves performance competitive with state-of-the-art egocentric domain generalization methods on both Argo1M and Ego4OOD, despite using fewer parameters and no additional modalities. Our empirical analysis demonstrates a clear relationship between measured covariate shift and recognition performance, highlighting the importance of controlled benchmarks and quantitative domain characterization for studying out-of-distribution generalization in egocentric video.

Problem

• egocentric video
• domain generalization
• covariate shift
• action recognition
• out-of-distribution

Innovation

• covariate shift
• domain generalization
• egocentric video
• binary decomposition
• OOD benchmark