Rethinking Complexity Metrics for LLM-Integrated Applications: Beyond Source Code

2026-07-02

AI Summary

This study addresses a critical gap in existing code complexity metrics, which overlook the runtime behavior dominated by natural language prompts in large language model (LLM) applications. The authors propose HECATE, a novel tool that treats prompts as behavioral specifications through a Prompt-as-Specification formalism, enabling joint assessment of complexity at both the prompt and code layers. From 25 dimensions, they derive 52 candidate metrics and empirically refine them using maintenance activities from version histories as proxy indicators, yielding 10 robust measures. Two top-performing metrics effectively predict maintenance effort across 20 components in six held-out repositories. Introducing new concepts such as structural breadth, the work identifies seven previously unknown prompt-layer complexity indicators, demonstrating that prompt complexity constitutes a key dimension independent of traditional code complexity.

Abstract

LLM-integrated applications blend natural language prompts with program code, and much of their runtime behavior originates in the prompt layer rather than in the code itself. Existing complexity metrics, however, operate solely at the code level and therefore overlook this behavioral logic entirely. We present HECATE, the first tool designed to assess complexity in both the prompt and code layers of such applications. Central to HECATE is Prompt-as-Specification, a Hoare-logic-inspired formalism that interprets every prompt as a specification of intended behavior. Grounded in 25 complexity dimensions identified across published taxonomies, the tool generates 52 candidate metrics. We assess each metric against 118 components collected from 18 open-source repositories, relying on maintenance activity derived from version history as an empirical proxy for complexity, and discard any metric that loses significance once code size is accounted for. Only ten metrics withstand this test. Seven belong to our newly introduced set; rather than measuring sheer volume, each tallies structurally distinct elements, such as LLM call sites, memory attributes, and prompt templates, an attribute we call structural breadth. Of the three surviving conventional metrics, RFC exhibits a similar breadth-oriented character, while Halstead N and V survive only as a residual effect of size; our top-performing metrics exceed all three. Crucially, the prompt-layer metrics retain significance even when the strongest code-level metric is added as a covariate, establishing prompt complexity as a dimension in its own right. A final validation on 20 components spanning six held-out repositories shows that the two best-performing metrics continue to predict maintenance effort, supporting their generalizability beyond the training set.

Problem

• LLM-integrated applications
• complexity metrics
• prompt layer
• code complexity
• maintenance effort

Innovation

• Prompt-as-Specification
• structural breadth
• LLM-integrated applications
• complexity metrics
• maintenance effort prediction