Beyond NL2Code: A Structured Survey of Multimodal Code Intelligence

2026-06-14

AI Summary

Existing code generation approaches primarily rely on textual descriptions and struggle to handle real-world programming tasks involving visual inputs such as screenshots, diagrams, or interactive states, which demand generated code to align with the visual reference in layout, geometry, semantics, and behavior. This work proposes a classification framework centered on “code roles” to systematically organize multimodal code intelligence research into four key domains: graphical user interfaces, scientific visualization, structured graphics, and emerging agent-based scenarios. By integrating multimodal perception, code generation, execution-based validation, and agent reasoning, the framework establishes a unified task and evaluation paradigm that clarifies how correctness evidence is handled across diverse settings, thereby laying a theoretical foundation and charting a path toward verifiable, executable code generation systems grounded in visual evidence.

Abstract

While LLMs have substantially advanced text-to-code synthesis, many real programming tasks specify intent through visual artifacts such as screenshots, charts, documents, vector drawings, videos, and interactive states. These tasks require models to connect visual perception to executable programs, because correctness depends not only on syntax but also on layout, geometry, data semantics, editability, interaction behavior, and domain-specific constraints that apply after execution. This survey examines Multimodal Code Intelligence, covering systems that generate, edit, refine, execute, or reason with code under visually grounded inputs and outputs. We first formulate the field by the role that code plays in each task, distinguishing code as a rendered artifact, an editable symbolic structure, a scientific representation, an intermediate reasoning trace, or an executable policy or tool interface. We then organize benchmarks and methods into four domains: Graphical User Interface, Scientific Visualization, Structured Graphics, and Frontier Tasks and Frameworks. This taxonomy connects mature artifact-generation problems to emerging agentic and unified settings and allows us to compare how different tasks treat evidence of correctness. Looking ahead, we argue that future research may benefit from four verification-centered directions. Multi-signal validation can combine complementary evidence of correctness, multi-state verification can test behavior across execution trajectories, cross-task transfer testing can probe reusable visual-code skills, and verifiable agent traces can reveal whether agent actions are grounded in visual evidence. Together, these directions may move multimodal code generation from single-output imitation toward evidence-grounded executable systems.

Problem

• Multimodal Code Intelligence
• Visual Programming
• Code Generation
• Executable Programs
• Visual Grounding

Innovation

• Multimodal Code Intelligence
• Visually Grounded Code Generation
• Verification-Centered Evaluation
• Cross-Task Transfer
• Executable Agent Traces