Learning to Assemble Novel Structures with Unfamiliar Parts under Semantic Constraints

2026-08-13

AI Summary

This study addresses the challenge of agents adapting to unknown semantic constraints and assembling novel structures post-deployment. We propose a neuro-symbolic architecture integrating natural language with visual observations, utilizing dialogue and demonstrations as symbolic evidence to facilitate online constraint acquisition and dynamic structure assembly. Experiments in a simulated truck assembly task demonstrate that language guidance significantly enhances online adaptive learning efficiency compared to methods relying solely on demonstrations or component naming. By validating the efficacy of neuro-symbolic reasoning in open-world structured tasks, this work establishes a novel paradigm for continuous agent learning, enabling robust adaptation to previously unseen semantic requirements through multimodal symbolic grounding.

Abstract

This paper describes a neurosymbolic architecture for learning to assemble novel structures using evidence from embodied conversations and task demonstrations. We focus on scenarios where an agent encounters, after deployment, semantic constraints on structures--in other words, constraints as to which part types and features make valid structures--that were not available during training, and where it is initially unaware of the relevant structure and component part concepts. The agent must acquire and exploit such knowledge through user interactions while attempting assembly. We study this setting in a simulated toy truck assembly domain, learning from symbolic evidence encoded in natural language and from dense visual observations. Our experiments show that communicating semantic constraints through natural language (e.g., "dump trucks have a dumper") yields more data-efficient online adaptation than relying only on task demonstrations and/or only naming the parts through natural language.

Problem

• Semantic Constraints
• Novel Structure Assembly
• Online Adaptation
• Embodied Conversations

Innovation

• Neurosymbolic Architecture
• Semantic Constraints
• Embodied Conversations
• Online Adaptation
• Novel Structure Assembly