Distilling On-device Language Models for Robot Planning with Minimal Human Intervention

2025-06-20

AI Summary

To address the challenge of deploying large language models (LLMs) on resource-constrained edge devices—particularly in communication-limited settings such as outdoor and industrial environments—for real-time robotic planning, this paper proposes PRISM: a fully automated, annotation-free framework for synthetic data generation and knowledge distillation. PRISM leverages instruction distillation, self-supervised synthetic data generation, and multi-task automatic evaluation and alignment to efficiently transfer knowledge from the teacher model (GPT-4o) to a lightweight student model (Llama-3.2-3B). Compared to baseline edge LLMs, PRISM achieves over 93% of GPT-4o’s planning performance—up from 10–20%—and significantly outperforms existing edge-adapted LLM approaches. The framework supports cross-platform deployment and zero-shot generalization across heterogeneous indoor/outdoor environments. Validated on both ground and aerial robots, PRISM delivers the first end-to-end deployable solution for edge intelligence in low-bandwidth, highly dynamic scenarios.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Intelligent Robots: Motion and Path Planning
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Large language models (LLMs) provide robots with powerful contextual reasoning abilities and a natural human interface. Yet, current LLM-enabled robots typically depend on cloud-hosted models, limiting their usability in environments with unreliable communication infrastructure, such as outdoor or industrial settings. We present PRISM, a framework for distilling small language model (SLM)-enabled robot planners that run on-device with minimal human supervision. Starting from an existing LLM-enabled planner, PRISM automatically synthesizes diverse tasks and environments, elicits plans from the LLM, and uses this synthetic dataset to distill a compact SLM as a drop-in replacement of the source model. We apply PRISM to three LLM-enabled planners for mapping and exploration, manipulation, and household assistance, and we demonstrate that PRISM improves the performance of Llama-3.2-3B from 10-20% of GPT-4o's performance to over 93% - using only synthetic data. We further demonstrate that the distilled planners generalize across heterogeneous robotic platforms (ground and aerial) and diverse environments (indoor and outdoor). We release all software, trained models, and datasets at https://zacravichandran.github.io/PRISM.

Problem

• Enable on-device robot planning without cloud dependency
• Distill small language models with minimal human supervision
• Improve performance across diverse robotic platforms and environments

Innovation

• Distills small on-device language models
• Automatically synthesizes diverse robotic tasks
• Improves performance using synthetic data