MM-Ego: Towards Building Egocentric Multimodal LLMs

2024-10-09

arXiv.org

AI Summary

Long-horizon (30-second to 1-hour), fine-grained question answering over egocentric videos suffers from a lack of high-quality benchmark data and robust evaluation protocols. Method: We propose MM-Ego, the first multimodal large language model tailored for egocentric video understanding. It introduces Memory Pointer Prompting—integrating global video summarization with keyframe retrieval—and employs a multi-stage video encoding framework with cross-modal alignment. A dedicated data engine automatically generates 7M high-quality QA samples. Contribution/Results: We release EgoBench, a rigorously de-biased benchmark comprising 629 egocentric videos and a standardized evaluation protocol. Extensive experiments demonstrate that MM-Ego significantly outperforms state-of-the-art methods across multiple egocentric video QA tasks, achieving breakthrough performance in long-horizon visual detail recognition and temporal memory retention.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

This research aims to comprehensively explore building a multimodal foundation model for egocentric video understanding. To achieve this goal, we work on three fronts. First, as there is a lack of QA data for egocentric video understanding, we develop a data engine that efficiently generates 7M high-quality QA samples for egocentric videos ranging from 30 seconds to one hour long, based on human-annotated data. This is currently the largest egocentric QA dataset. Second, we contribute a challenging egocentric QA benchmark with 629 videos and 7,026 questions to evaluate the models' ability in recognizing and memorizing visual details across videos of varying lengths. We introduce a new de-biasing evaluation method to help mitigate the unavoidable language bias present in the models being evaluated. Third, we propose a specialized multimodal architecture featuring a novel"Memory Pointer Prompting"mechanism. This design includes a global glimpse step to gain an overarching understanding of the entire video and identify key visual information, followed by a fallback step that utilizes the key visual information to generate responses. This enables the model to more effectively comprehend extended video content. With the data, benchmark, and model, we successfully build MM-Ego, an egocentric multimodal LLM that shows powerful performance on egocentric video understanding.

Problem

• Building multimodal foundation model for egocentric video understanding
• Automatically generating large-scale QA dataset for egocentric videos
• Proposing specialized architecture with Memory Pointer Prompting mechanism

Innovation

• Generated 7M QA samples automatically
• Introduced de-biasing evaluation method
• Proposed Memory Pointer Prompting mechanism