Acquiring Grounded Representations of Words with Situated Interactive Instruction

2025-02-28

AI Summary

This study addresses the challenge of jointly acquiring perceptual, semantic, and procedural knowledge in embodied lexical learning. We propose a human-robot hybrid active interaction framework enabling a robot to dynamically identify unknown concepts in real-world desktop environments, autonomously initiate contextualized learning requests to human teachers, and unify multimodal knowledge acquisition within a “perception–comprehension–planning–execution” semantic closed loop. The system is built upon the Soar cognitive architecture and integrates visual perception, natural language understanding, task planning, and robotic arm control. Experiments demonstrate significant improvements in rapid generalization to novel lexical items and instruction execution accuracy, along with zero-shot concept transfer capability. Our core contributions are (1) the first hybrid active interaction mechanism for embodied lexical learning, and (2) a unified representational paradigm for heterogeneous knowledge types—perceptual, semantic, and procedural—within a single cognitive architecture.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Natural Language Processing: Lexical Semantics and Morphology

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

We present an approach for acquiring grounded representations of words from mixed-initiative, situated interactions with a human instructor. The work focuses on the acquisition of diverse types of knowledge including perceptual, semantic, and procedural knowledge along with learning grounded meanings. Interactive learning allows the agent to control its learning by requesting instructions about unknown concepts, making learning efficient. Our approach has been instantiated in Soar and has been evaluated on a table-top robotic arm capable of manipulating small objects.

Problem

• Acquiring grounded word representations through interactive learning
• Focusing on perceptual, semantic, and procedural knowledge acquisition
• Implementing approach in Soar for robotic arm object manipulation

Innovation

• Interactive learning with human instructors
• Acquisition of perceptual, semantic, procedural knowledge
• Implemented in Soar for robotic arm manipulation