Exploring Direct Instruction and Summary-Mediated Prompting in LLM-Assisted Code Modification

2025-08-02

AI Summary

Existing prompt engineering for LLM-assisted code modification lacks systematic analysis of strategy trade-offs, distinct from code generation tasks. Method: We propose and comparatively evaluate two prompting strategies—Direct Instruction Prompting (emphasizing flexibility and efficiency) and Summary-Mediated Prompting (leveraging natural-language summaries to enhance semantic alignment, comprehension controllability, and modification accuracy)—within a unified framework integrating summary generation, semantic reasoning, and iterative human-AI collaboration. Contribution/Results: A 15-developer empirical study reveals that practitioners dynamically select strategies based on task objectives: Summary-Mediated Prompting significantly improves understandability and controllability for complex modifications, whereas Direct Instruction excels for lightweight, high-frequency edits. This work is the first to systematically characterize the flexibility–comprehensibility–control trade-off between these strategies, establishing a scalable, task-adaptive prompting paradigm for interactive code modification.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

This paper presents a study of using large language models (LLMs) in modifying existing code. While LLMs for generating code have been widely studied, their role in code modification remains less understood. Although "prompting" serves as the primary interface for developers to communicate intents to LLMs, constructing effective prompts for code modification introduces challenges different from generation. Prior work suggests that natural language summaries may help scaffold this process, yet such approaches have been validated primarily in narrow domains like SQL rewriting. This study investigates two prompting strategies for LLM-assisted code modification: Direct Instruction Prompting, where developers describe changes explicitly in free-form language, and Summary-Mediated Prompting, where changes are made by editing the generated summaries of the code. We conducted an exploratory study with 15 developers who completed modification tasks using both techniques across multiple scenarios. Our findings suggest that developers followed an iterative workflow: understanding the code, localizing the edit, and validating outputs through execution or semantic reasoning. Each prompting strategy presented trade-offs: direct instruction prompting was more flexible and easier to specify, while summary-mediated prompting supported comprehension, prompt scaffolding, and control. Developers' choice of strategy was shaped by task goals and context, including urgency, maintainability, learning intent, and code familiarity. These findings highlight the need for more usable prompt interactions, including adjustable summary granularity, reliable summary-code traceability, and consistency in generated summaries.

Problem

• Investigating effective prompting strategies for LLM-assisted code modification
• Comparing Direct Instruction vs Summary-Mediated Prompting for code changes
• Understanding developer workflows and trade-offs in LLM-based code editing

Innovation

• Direct Instruction Prompting for explicit code changes
• Summary-Mediated Prompting via editing generated summaries
• Iterative workflow with comprehension and validation