Evaluating Gemma4 Models as AI Teaching Assistants for Introductory Parallel Programming: A DataRaceBench Study

2026-06-12

AI Summary

This study addresses the challenges students face in debugging data races in parallel programming due to concurrency nondeterminism and the complexity of shared-memory semantics. It presents the first systematic evaluation of low-cost open-source large language models—specifically Gemma4-4B and Gemma4-31B—on their ability to identify, explain, and repair data races in OpenMP programs using the DataRaceBench benchmark. The work investigates the impact of contextual prompts, including ThreadSanitizer reports and model-generated explanations, on repair efficacy. Experimental results demonstrate that model scale significantly influences performance: Gemma4-31B correctly explains 100 cases and repairs 98, substantially outperforming Gemma4-4B (82 explained, 73 repaired). Notably, additional contextual prompts do not consistently improve repair outcomes, highlighting the practical potential of large models in educational settings for parallel debugging.

Abstract

Debugging data races is a major challenge for students learning parallel programming due to the non-deterministic nature of concurrent execution and the complexity of shared-memory semantics. Recent advances in Large Language Models (LLMs) suggest that they could serve as AI teaching assistants, but the capabilities of lower-cost open-weight models for parallel debugging remain unclear. In this paper, we evaluate two Gemma4 open-weight models, Gemma4-E4B and Gemma4-31B, on their ability to identify, explain, and repair data races in OpenMP programs from the DataRaceBench benchmark suite. We also investigate whether contextual hints, including ThreadSanitizer (TSan) reports and model-generated explanations, improve repair quality. Our results show that Gemma4-E4B correctly explained 82 of 104 race-condition programs and successfully repaired 73, while Gemma4-31B achieved 100 correct explanations and 98 successful repairs. Surprisingly, additional context did not consistently improve repair effectiveness and sometimes reduced performance. These findings suggest that open-weight LLMs can provide valuable support for student self-debugging, with larger models offering near-complete coverage of the benchmark suite.

Problem

• data races
• parallel programming
• debugging
• concurrent execution
• shared-memory semantics

Innovation

• Large Language Models
• Parallel Programming Education
• Data Race Debugging
• Open-Weight Models
• DataRaceBench