Generative AI in Systems Engineering: A Framework for Risk Assessment of Large Language Models

2026-02-04

AI Summary

This work proposes the LLM Risk Framework (LRF), a novel approach to structuring risk assessment for large language model (LLM) integration in systems engineering. Addressing the lack of standardized methodologies that often leads to ad hoc deployment, ambiguous failure modes, and limited scalability, the LRF uniquely integrates systems engineering principles with generative AI governance. It introduces a two-dimensional classification scheme based on autonomy—ranging from assistive to fully autonomous decision-making—and impact severity, defined by the potential harm of erroneous outputs on engineering processes. This framework enables systematic identification of risk levels and informs tailored validation strategies, human oversight requirements, and mitigation measures. By doing so, it facilitates the safe, transparent, and controllable incorporation of LLMs into complex engineering environments, providing an actionable foundation for trustworthy AI deployment and the development of assurance standards.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

The increasing use of Large Language Models (LLMs) offers significant opportunities across the engineering lifecycle, including requirements engineering, software development, process optimization, and decision support. Despite this potential, organizations face substantial challenges in assessing the risks associated with LLM use, resulting in inconsistent integration, unknown failure modes, and limited scalability. This paper introduces the LLM Risk Assessment Framework (LRF), a structured approach for evaluating the application of LLMs within Systems Engineering (SE) environments. The framework classifies LLM-based applications along two fundamental dimensions: autonomy, ranging from supportive assistance to fully automated decision making, and impact, reflecting the potential severity of incorrect or misleading model outputs on engineering processes and system elements. By combining these dimensions, the LRF enables consistent determination of corresponding risk levels across the development lifecycle. The resulting classification supports organizations in identifying appropriate validation strategies, levels of human oversight, and required countermeasures to ensure safe and transparent deployment. The framework thereby helps align the rapid evolution of AI technologies with established engineering principles of reliability, traceability, and controlled process integration. Overall, the LRF provides a basis for risk-aware adoption of LLMs in complex engineering environments and represents a first step toward standardized AI assurance practices in systems engineering.

Problem

• Large Language Models
• Risk Assessment
• Systems Engineering
• AI Assurance
• Model Autonomy

Innovation

• Large Language Models
• Risk Assessment Framework
• Systems Engineering
• Autonomy-Impact Classification
• AI Assurance