Auto-DSM Under the Lens: A Black-Box Evaluation Framework for LLM-Based DSM Generation

2026-07-07

AI Summary

This study evaluates the performance and reliability of closed-source large language models (LLMs) in automatically generating Design Structure Matrices (DSMs) from technical documentation. To this end, it introduces the first reproducible black-box evaluation framework tailored for LLM-generated DSMs, integrating structural metrics (completeness, correctness, coupling density), classification metrics (selective precision, abstention coverage), and stability measures (entropy, Fleiss’ κ). A Composite Quality Score is further proposed to enable transparent auditing. Experimental results demonstrate that LLMs can produce highly reproducible DSMs when provided with structured inputs; however, their outputs remain sensitive to ambiguity, inconsistent dependency definitions, and variations in prompt phrasing. These findings illuminate both the promise and current limitations of automated DSM generation using LLMs.

Abstract

This paper presents a black-box evaluation framework to systematically assess the ability of Large Language Models (LLMs) to generate Design Structure Matrices (DSMs) from structured technical documentation. Motivated by the closed-source nature of current Auto-DSM pipelines, the framework introduces a reproducible methodology that benchmarks generated DSMs (GEN-DSMs) against manually validated ground-truth matrices (GT-DSMs). The evaluation integrates both single-run and multi-run perspectives, combining structural metrics (Completeness, Correctness, Coupling Density), classification metrics (Selective Accuracy, Abstention Coverage), and stability measures (Entropy, Fleiss' $κ$). To synthesize these aspects, a Composite Quality Score (Q) is proposed. Controlled experiments are conducted on two datasets: a fictive abstract system and a real-world refrigerator decomposition, covering variations in phrasing, parameter-dataset alignment, and system complexity. Results show that LLMs can produce structurally plausible DSMs and achieve high reproducibility under well-structured inputs, but remain sensitive to ambiguity, inconsistent dependency definitions, and prompt formulation. The findings highlight systematic sources of hallucination and abstention failure, demonstrating both the potential and current limitations of LLM-driven DSM automation. The proposed framework provides a transparent benchmark for auditing Auto-DSM pipelines and establishes foundations for integrating LLM-based decomposition methods into model-based systems engineering (MBSE) workflows.

Problem

• Design Structure Matrix
• Large Language Models
• Auto-DSM
• Black-box Evaluation
• Model-based Systems Engineering

Innovation

• Auto-DSM
• Large Language Models
• Design Structure Matrix
• Black-box Evaluation
• Composite Quality Score