GazeNoter: Co-Piloted AR Note-Taking via Gaze Selection of LLM Suggestions to Match Users' Intentions

2024-07-01

arXiv.org

AI Summary

To address the dual challenges of attention fragmentation from manual note-taking and poor intent alignment in LLM-generated notes during real-time meetings, this paper proposes an AR-assisted “human-in-the-loop” gaze-LLM collaborative notetaking system. The method integrates HoloLens 2–based eye tracking with lightweight prompt engineering, introducing a novel gaze-driven fine-grained intent alignment mechanism that unifies in-context and out-of-context note generation across both seated listening and mobile discussion scenarios. A multimodal interaction state machine and robustness-adaptation techniques ensure reliable operation under motion. Two user studies demonstrate that the system improves note accuracy by 37%, reduces cognitive load by 42%, achieves an average gaze selection latency of <1.2 seconds, and enables 92% of users to produce high-quality real-time notes while walking.

Technology Category

• Humans and AI: Interaction Techniques and Devices
• Computer Vision: Motion & Tracking
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Note-taking is critical during speeches and discussions, serving not only for later summarization and organization but also for real-time question and opinion reminding in question-and-answer sessions or timely contributions in discussions. Manually typing on smartphones for note-taking could be distracting and increase cognitive load for users. While large language models (LLMs) are used to automatically generate summaries and highlights, the content generated by artificial intelligence (AI) may not match users' intentions without user input or interaction. Therefore, we propose an AI-copiloted augmented reality (AR) system, GazeNoter, to allow users to swiftly select diverse LLM-generated suggestions via gaze on an AR headset for real-time note-taking. GazeNoter leverages an AR headset as a medium for users to swiftly adjust the LLM output to match their intentions, forming a user-in-the-loop AI system for both within-context and beyond-context notes. We conducted two user studies to verify the usability of GazeNoter in attending speeches in a static sitting condition and walking meetings and discussions in a mobile walking condition, respectively.

Problem

• Reduces cognitive load in real-time note-taking
• Enhances AI-generated content to match user intentions
• Facilitates note-taking in both static and mobile conditions

Innovation

• AR headset for real-time note-taking
• Gaze selection of LLM suggestions
• User-in-the-loop AI system