Task Planning for Mobile Manipulation in Retail Stores using Foundation Models with Iterative Re-planning

2026-07-10

AI Summary

This work addresses the challenge of automated task planning for product restocking in the unstructured environments of retail stores by proposing an end-to-end planning framework that integrates large language models (LLMs) with vision-language models (VLMs). The system leverages user prompts and employs a feedback-driven iterative replanning mechanism to enable robust task execution and online error correction on an omnidirectional mobile manipulation platform. To the best of our knowledge, this is the first study to combine foundation models with iterative replanning specifically for retail shelf restocking. The approach demonstrates high efficiency and adaptability in performing pick-and-place tasks within dynamic, human-centric scenarios, as validated in PyBullet simulation environments.

Abstract

Automation in industries such as retail, warehousing and logistics presents opportunities for greater throughput, cost reduction and mitigation of disruptions from labour shortages. Previously, such efforts have focused on back-room operations involving packing and sorting in relatively structured environments. With advances in robotic mobile manipulation hardware and foundation models, automation can now be applied to more variable and human-centric environments such as retail store shelves. In this work, we present a task-planning approach using Large Language Models (LLMs) and Vision-Language Models (VLMs) to address the restocking problem in retail scenarios such as supermarkets. We demonstrate this system on a custom omnidirectional mobile manipulation platform, with user-driven prompts and a feedback-based iterative re-planning approach for error correction. The end-to-end system is validated in a PyBullet simulation environment for pick-and-place tasks.

Problem

• retail automation
• mobile manipulation
• restocking
• task planning
• foundation models

Innovation

• Foundation Models
• Iterative Re-planning
• Mobile Manipulation
• Retail Automation
• Vision-Language Models