OdinEval: A Reproducible Benchmark for LLM-Based Program Repair in the Odin Programming Language

2026-08-19

AI Summary

为了解决Odin语言中程序修复基准不足的问题，研究构建了OdinEval基准，通过文档化的缺陷和特定测试来评估六种语言模型在168个实例上的修复性能。

Abstract

Repository-level repair benchmarks still center on a few mainstream languages, leaving systems languages such as Odin largely untested. We present OdinEval, a reproducible benchmark built from documented defects in public Odin repositories. Each instance binds an issue to base and fix commits, a gold patch, an issue-specific regression test, a historical toolchain, and execution records. Admission requires the test to fail on the base revision and pass after the gold fix. When no usable developer test exists, a black-box test is reviewed independently by three instances of the same model, executed in both historical states, and revised from recorded feedback under a versioned Test Writing Skill. We evaluate six language models on 168 filtered instances under one shared protocol. Kimi-K3 records the highest Resolved score at 66.7%, while Qwen3.8-Max has the highest Repro score at 96.4%. The release includes frozen data, source archives, containers, validators, model patches, and audit manifests.

Problem

• Odin Programming Language
• Program Repair
• Benchmark
• Reproducibility
• Systems Languages

Innovation

• Reproducible Benchmark
• Program Repair
• Odin Programming Language
• Black-Box Test
• Test Writing Skill