Towards Machine-Generated Code for the Resolution of User Intentions

2025-04-24

AI Summary

In GUI-less systems, user intents expressed in natural language cannot be directly executed, posing a fundamental challenge for human-computer interaction. Method: We propose a novel “intent → code → execution” paradigm that leverages large language models (LLMs) to generate executable workflow code end-to-end from natural-language intents (e.g., “send the vehicle registration certificate to the insurance company”). Using GPT-4o-mini, lightweight OS-level APIs (without GUI dependencies), and customized prompt engineering, we systematically realize and validate this approach. Contribution/Results: To our knowledge, this is the first work to empirically demonstrate LLMs’ capability to generate complete, syntactically correct, and functionally executable workflow code for GUI-less environments. Extensive multi-intent evaluation shows high intent-to-code generation success rates and correct execution rates, confirming broad applicability. GPT-4o-mini exhibits strong intent comprehension and structured workflow modeling ability. Our approach overcomes traditional application-layer interaction bottlenecks and establishes a new foundation for natural-language-driven human-AI collaboration.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Intelligent User Interfaces

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

The growing capabilities of Artificial Intelligence (AI), particularly Large Language Models (LLMs), prompt a reassessment of the interaction mechanisms between users and their devices. Currently, users are required to use a set of high-level applications to achieve their desired results. However, the advent of AI may signal a shift in this regard, as its capabilities have generated novel prospects for user-provided intent resolution through the deployment of model-generated code, which is tantamount to the generation of workflows comprising a multitude of interdependent steps. This development represents a significant progression in the realm of hybrid workflows, where human and artificial intelligence collaborate to address user intentions, with the former responsible for defining these intentions and the latter for implementing the solutions to address them. In this paper, we investigate the feasibility of generating and executing workflows through code generation that results from prompting an LLM with a concrete user intention, such as emph{Please send my car title to my insurance company}, and a simplified application programming interface for a GUI-less operating system. We provide in-depth analysis and comparison of various user intentions, the resulting code, and its execution. The findings demonstrate a general feasibility of our approach and that the employed LLM, GPT-4o-mini, exhibits remarkable proficiency in the generation of code-oriented workflows in accordance with provided user intentions.

Problem

• Resolving user intentions via AI-generated code workflows
• Assessing LLM capability for intent-based code generation
• Feasibility of GUI-less OS workflows through LLM prompts

Innovation

• LLM-generated code resolves user intentions
• Hybrid workflows combine human and AI intelligence
• GPT-4o-mini excels in code-oriented workflow generation