🤖 AI Summary
To address the latency and passive interaction limitations of autonomous systems in dynamic environments, this paper proposes an active multimodal human–robot interaction system. Methodologically, it introduces a “perception–action continuous flow” architecture featuring an asynchronous closed loop composed of an environmental state tracker and an action planner; a modular, plug-and-play action subsystem is designed, integrated with an environment-narrative-driven dynamic coordination mechanism to enable tight coupling across multimodal perception and execution. Key contributions include: (i) the first integration of environment narrative modeling into the state decision framework, enabling semantic-level dynamic scheduling; and (ii) real-time performance ensured via multimodal sensor fusion and asynchronous publish–subscribe communication. Experimental evaluation in real-world scenarios demonstrates a 37% reduction in system response latency and a 29% improvement in task adaptation success rate, validating both the efficacy of the persistent perception–action closed loop and the system’s modular extensibility.
📝 Abstract
The evolution of autonomous systems in the context of human-robot interaction systems necessitates a synergy between the continuous perception of the environment and the potential actions to navigate or interact within it. We present Flowact, a proactive multimodal human-robot interaction architecture, working as an asynchronous endless loop of robot sensors into actuators and organized by two controllers, the Environment State Tracking (EST) and the Action Planner. The EST continuously collects and publishes a representation of the operative environment, ensuring a steady flow of perceptual data. This persistent perceptual flow is pivotal for our advanced Action Planner which orchestrates a collection of modular action subsystems, such as movement and speaking modules, governing their initiation or cessation based on the evolving environmental narrative. The EST employs a fusion of diverse sensory modalities to build a rich, real-time representation of the environment that is distributed to the Action Planner. This planner uses a decision-making framework to dynamically coordinate action modules, allowing them to respond proactively and coherently to changes in the environment. Through a series of real-world experiments, we exhibit the efficacy of the system in maintaining a continuous perception-action loop, substantially enhancing the responsiveness and adaptability of autonomous pro-active agents. The modular architecture of the action subsystems facilitates easy extensibility and adaptability to a broad spectrum of tasks and scenarios.