Benchmarking Large Language Models on Floating-Point Error Classification

2026-06-30

AI Summary

This study addresses the limited capability of large language models (LLMs) in statically detecting floating-point errors in code by introducing InterFLOPBench, the first specialized benchmark comprising 90 C-language kernels and 1,130 test cases. The task is formulated as a multi-label classification problem encompassing six canonical categories of floating-point errors. The authors propose an evaluation framework based on multi-label F1-score and conduct a systematic assessment across 14 prominent LLMs. Results show that state-of-the-art models, such as Qwen3-32B and Gemini 2.5 Flash, achieve overall F1-scores exceeding 0.88, with explicit errors like division-by-zero detected at a rate of 0.8479. However, implicit errors—including underflow and cancellation—remain challenging, with recognition rates around 0.60–0.62, highlighting current limitations in LLMs’ understanding of complex numerical semantics.

Abstract

This paper investigates the capability of Large Language Models (LLMs) to detect and classify floating-point errors statically in software code. We introduce InterFLOPBench, a benchmark of 90 C kernels with 1 130 test samples designed to evaluate LLMs across six categories of floating-point error: cancellation, comparison, division by zero, overflow, underflow and NaN, compared across 14 LLMs. The evaluation framework treats floating-point error detection as a multi-label classification problem and employs the F1-score metric to measure performance. Results demonstrate that latest models (Qwen 3 32b, Gemini 2.5 Flash, Phi 4 Reasoning, DeepSeek R1T2, and gpt-oss 20b and 120b) achieve a performance greater than 0.88 overall F1-score. Performance varies between error categories, between explicit operations such as division by zero (Average F1-score: 0.8479) and more subtle numerical phenomena such as underflow (Average F1-score: 0.6059) and cancellation (Average F1-score: 0.6164).

Problem

• Large Language Models
• floating-point errors
• error classification
• static analysis
• benchmarking

Innovation

• Large Language Models
• Floating-Point Error Classification
• InterFLOPBench
• Static Code Analysis
• Multi-label Classification