UNIEGO: Proxies as Mediators for Unified Egocentric Video Representation Learning

2026-06-18

AI Summary

Existing wearable cameras struggle to comprehensively model human behavior due to their limited field of view and single modality. This work proposes a hierarchical multi-teacher distillation framework that unifies knowledge from multi-view (ego-exo), multi-modal (RGB, depth, skeleton), and foundation models into an egocentric representation space through modality-specific proxy models. The approach innovatively introduces a selective proxy distillation mechanism and an initialization strategy based on convex combinations of proxy parameters, enabling adaptive selection of reliable supervision signals to enhance training stability and efficiency. The method achieves state-of-the-art performance across three tasks—action recognition, video retrieval, and action segmentation—significantly outperforming naive multi-teacher distillation baselines.

Abstract

Egocentric video understanding is inherently limited by the narrow perspective of wearable cameras: a single viewpoint, a single modality, a single model cannot capture the full richness of human action. We argue that a truly expressive egocentric representation must subsume complementary knowledge across viewpoints, modalities, and foundation model representations, yet remain deployable from egocentric video alone. To this end, we introduce a hierarchical multi-teacher distillation framework that produces UNIEGO, a unified egocentric encoder trained with nine teachers spanning ego-exo viewpoints, RGB, depth, and skeleton modalities, and four foundation models. Rather than distilling directly from heterogeneous teachers whose incompatible architectures and feature geometries induce conflicting gradients, our framework interposes a layer of representation-specific Proxy models that translate diverse teacher knowledge into a homogeneous egocentric space. A second distillation stage, Selective Proxy Distillation (SPD), then adaptively selects, for each training sample, the subset of proxies that are both correct and confident, distilling exclusively from reliable supervision and suppressing erroneous signals. SPD is further stabilized by initializing UNIEGO as a learned convex combination of proxy parameters, placing the unified model in a well-conditioned region of the loss landscape before distillation begins. UNIEGO achieves state-of-the-art performance across three egocentric video understanding tasks - action recognition, video retrieval, and action segmentation on three challenging ego-exo benchmarks, outperforming naive multi-teacher distillation baselines and demonstrating that structured, proxy-mediated knowledge transfer yields richer and more discriminative egocentric representations.

Problem

• egocentric video understanding
• multi-viewpoint learning
• multimodal representation
• unified representation learning
• knowledge distillation

Innovation

• proxy-mediated distillation
• egocentric video representation
• multi-teacher knowledge distillation
• selective proxy distillation
• unified representation learning