Vibe Coding on Trial: Operating Characteristics of Unanimous LLM Juries

2026-02-02

International Computer Science Conference

AI Summary

研究使用LLM评审委员会方法，通过构建一致同意的模型小组来评估代码安全性，以减少错误接受率并提高查询通过的安全性。

Abstract

Large Language Models (LLMs) are now good enough at coding that developers can describe intent in plain language and let the tool produce the first code draft, a workflow increasingly built into tools like GitHub Copilot, Cursor, and Replit. What is missing is a reliable way to tell which model written queries are safe to accept without sending everything to a human. We study the application of an LLM jury to run this review step. We first benchmark 15 open models on 82 MySQL text to SQL tasks using an execution grounded protocol to get a clean baseline of which models are strong. From the six best models we build unanimous committees of sizes 1 through 6 that see the prompt, schema, and candidate SQL and accept it only when every member says it is correct. This rule matches safety first deployments where false accepts are more costly than false rejects. We measure true positive rate, false positive rate and Youden J and we also look at committees per generator. Our results show that single model judges are uneven, that small unanimous committees of strong models can cut false accepts while still passing many good queries, and that the exact committee composition matters significantly.

Problem

• Large Language Models
• code generation
• query safety
• automated review

Innovation

• LLM jury
• unanimous committees
• code safety
• execution grounded protocol