Beyond Code Generation: LLM-supported Exploration of the Program Design Space

2025-03-10

AI Summary

This paper addresses the narrow design-space exploration and severe information overload in LLM-assisted programming by proposing an IDE framework deeply integrated into the design process. Methodologically, it elevates LLMs from single-point code generators to collaborative design-space exploration agents—a novel conceptual shift—explicitly modeling problem variants, solution pathways, and implicit assumptions via a design decision graph, and enabling multi-perspective problem reformulation and parallel solution generation. Contributions include: (1) establishing the first LLM-IDE co-design paradigm specifically for program design-space exploration; (2) enabling design decisions to be traceable, comparable, and evolvable; and (3) demonstrating through user studies a statistically significant increase in design breadth. The work further identifies attention management as the core bottleneck in LLM-augmented IDEs, thereby providing a new benchmark for human-AI collaborative design research.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Humans and AI: Game Design — Procedural Content Generation & Storytelling

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

In this work, we explore explicit Large Language Model (LLM)-powered support for the iterative design of computer programs. Program design, like other design activity, is characterized by navigating a space of alternative problem formulations and associated solutions in an iterative fashion. LLMs are potentially powerful tools in helping this exploration; however, by default, code-generation LLMs deliver code that represents a particular point solution. This obscures the larger space of possible alternatives, many of which might be preferable to the LLM's default interpretation and its generated code. We contribute an IDE that supports program design through generating and showing new ways to frame problems alongside alternative solutions, tracking design decisions, and identifying implicit decisions made by either the programmer or the LLM. In a user study, we find that with our IDE, users combine and parallelize design phases to explore a broader design space -- but also struggle to keep up with LLM-originated changes to code and other information overload. These findings suggest a core challenge for future IDEs that support program design through higher-level instructions given to LLM-based agents: carefully managing attention and deciding what information agents should surface to program designers and when.

Problem

• Explores LLM support for iterative program design.
• Addresses limitations of code-generation LLMs in revealing alternatives.
• Proposes IDE for managing design space and decision tracking.

Innovation

• LLM-powered IDE for program design exploration
• Generates alternative problem formulations and solutions
• Tracks design decisions and identifies implicit choices