Is Vibe Coding the Future? An Empirical Assessment of LLM Generated Codes for Construction Safety

2026-04-14

AI Summary

This study systematically evaluates the reliability of “vibe coding” in architectural safety applications, with a focus on the risk of “silent failures”—code that executes successfully but contains critical logical errors. Leveraging a role-driven prompt dataset and a dual-channel evaluation framework combining dynamic sandboxed execution with LLM-as-a-Judge assessment, the authors conduct an empirical analysis of 450 Python scripts generated by Claude 3.5 Haiku, GPT-4o-Mini, and Gemini 2.5 Flash. The work presents the first quantification of silent failure rates in high-stakes engineering contexts (approximately 45% overall), reveals a significant association between user role specification and model hallucination, and identifies a 56% rate of mathematical errors in functionally executable code produced by GPT-4o-Mini. Despite an ~85% syntactic execution success rate, pervasive logical flaws underscore the urgent need for deterministic AI wrappers and stringent governance mechanisms.

Technology Category

• Natural Language Processing: Safety and Robustness
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Large-scale security measurements

Abstract

The emergence of vibe coding, a paradigm where non-technical users instruct Large Language Models (LLMs) to generate executable codes via natural language, presents both significant opportunities and severe risks for the construction industry. While empowering construction personnel such as the safety managers, foremen, and workers to develop tools and software, the probabilistic nature of LLMs introduces the threat of silent failures, wherein generated code compiles perfectly but executes flawed mathematical safety logic. This study empirically evaluates the reliability, software architecture, and domain-specific safety fidelity of 450 vibe-coded Python scripts generated by three frontier models, Claude 3.5 Haiku, GPT-4o-Mini, and Gemini 2.5 Flash. Utilizing a persona-driven prompt dataset (n=150) and a bifurcated evaluation pipeline comprising isolated dynamic sandboxing and an LLM-as-a-Judge, the research quantifies the severe limits of zero-shot vibe codes for construction safety. The findings reveal a highly significant relationship between user persona and data hallucination, demonstrating that less formal prompts drastically increase the AI's propensity to invent missing safety variables. Furthermore, while the models demonstrated high foundational execution viability (~85%), this syntactic reliability actively masked logic deficits and a severe lack of defensive programming. Among successfully executed scripts, the study identified an alarming ~45% overall Silent Failure Rate, with GPT-4o-Mini generating mathematically inaccurate outputs in ~56% of its functional code. The results demonstrate that current LLMs lack the deterministic rigor required for standalone safety engineering, necessitating the adoption of deterministic AI wrappers and strict governance for cyber-physical deployments.

Problem

• vibe coding
• Large Language Models
• construction safety
• silent failure
• code reliability

Innovation

• vibe coding
• silent failure
• construction safety
• LLM hallucination
• deterministic AI wrapper