From Pixels to Shelf: End-to-End Algorithmic Control of a Mobile Manipulator for Supermarket Stocking and Fronting

2025-09-15

AI Summary

To address the challenges of dynamic human–robot interaction, spatial constraints, and geometric diversity of retail items in supermarket shelf replenishment and restocking, this paper proposes an end-to-end autonomous robotic system. Built on ROS2, the system integrates behavior-tree-based task planning, a fine-tuned lightweight vision detection model, and ArUco marker–guided two-stage model predictive control (MPC) to realize a closed-loop perception–planning–control architecture. Key contributions include: (1) robust, behavior-tree-driven task scheduling; (2) efficient domain-adaptive fine-tuning of vision models tailored to retail environments; and (3) high-precision grasp-and-place control achieved by tightly coupling visual localization with MPC. Evaluated in a simulated supermarket environment, the system successfully completed over 700 replenishment tasks, achieving >98% success rates for both item pickup and placement—demonstrating its reliability and deployability in real-world retail settings.

Technology Category

• Intelligent Robots: Manipulation
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Computer Vision: Vision for Robotics & Autonomous Driving

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Ad search and search for Web retail
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Autonomous stocking in retail environments, particularly supermarkets, presents challenges due to dynamic human interactions, constrained spaces, and diverse product geometries. This paper introduces an efficient end-to-end robotic system for autonomous shelf stocking and fronting, integrating commercially available hardware with a scalable algorithmic architecture. A major contribution of this work is the system integration of off-the-shelf hardware and ROS2-based perception, planning, and control into a single deployable platform for retail environments. Our solution leverages Behavior Trees (BTs) for task planning, fine-tuned vision models for object detection, and a two-step Model Predictive Control (MPC) framework for precise shelf navigation using ArUco markers. Laboratory experiments replicating realistic supermarket conditions demonstrate reliable performance, achieving over 98% success in pick-and-place operations across a total of more than 700 stocking events. However, our comparative benchmarks indicate that the performance and cost-effectiveness of current autonomous systems remain inferior to that of human workers, which we use to highlight key improvement areas and quantify the progress still required before widespread commercial deployment can realistically be achieved.

Problem

• Develops autonomous robotic system for supermarket stocking
• Integrates perception, planning, control for shelf manipulation
• Addresses dynamic human interactions and product diversity challenges

Innovation

• Integrates off-the-shelf hardware with ROS2-based architecture
• Uses Behavior Trees for task planning and fine-tuned vision
• Implements two-step MPC framework for precise shelf navigation