ASCII Art Turns LLMs into VLA Controllers

2026-06-19

AI Summary

This work addresses the high computational cost and deployment challenges of conventional vision-language-action (VLA) controllers that rely on multimodal large models. The authors propose encoding visual observations into ASCII art representations, enabling a purely text-based large language model (LLM) to perceive the environment, interpret natural language instructions, and generate executable action sequences without any dedicated visual modules. This approach establishes a novel, lightweight, and interpretable paradigm for bridging visual inputs to textual reasoning, demonstrating for the first time that a text-only LLM can effectively perform VLA control through ASCII-based scene representation. Experimental results show that, when fine-tuned with expert demonstrations and DAgger, the LLM accurately identifies targets and plans effective actions in 2D manipulation tasks, achieving performance on par with traditional VLA methods both in simulation and on a real robotic arm.

Abstract

Vision--Language--Action (VLA) controllers are often built by extending vision--language models (VLMs) with action supervision, relying on multimodal backbones with large data and compute requirements. We demonstrate that a text-only large language model (LLM) can be adapted into a VLA-style controller when visual observations are rendered into a text input using an ASCII representation. This ASCII-as-vision interface enables existing training and deployment stacks for LLMs to efficiently condition on visual state, follow natural-language instructions, and produce constrained, executable actions. We fine-tune and compare multiple LLMs and VLMs across model families and scales, using both expert demonstrations from a planning-based teacher, as well as DAgger for iterative improvement. In a 2D manipulation benchmark, in both simulation and on a physical manipulator, the resulting controllers can identify task-relevant entities and plan feasible action sequences. Our results suggest that ASCII rendering can serve as a lightweight, interpretable modality bridge from images to text, complementing conventional VLA pipelines, and opening directions for VLA research with text-only backbones.

Problem

• Vision-Language-Action
• ASCII Art
• Large Language Models
• Visual Reasoning
• Robot Control

Innovation

• ASCII Art
• Large Language Models
• Vision-Language-Action
• Modality Bridge
• Text-only Backbone