🤖 AI Summary
In regulated domains such as finance and healthcare, industrial agents must perform arithmetic reasoning over structured data locally while ensuring sensitive information remains within the organizational boundary—posing challenges for accuracy, privacy, interpretability, and parameter efficiency.
Method: We propose an error-driven prompt optimization framework that automatically clusters model prediction errors, iteratively refines interpretable prompt rules, and requires no parameter fine-tuning. Integrating error analysis, hierarchical clustering, and dynamic prompt engineering, the method establishes an end-to-end arithmetic reasoning enhancement pipeline on the Qwen3-4B small language model.
Contribution/Results: Under strict local deployment constraints, our approach achieves 70.8% arithmetic reasoning accuracy—significantly outperforming GPT-3.5 Turbo—and marks the first solution enabling high-accuracy, privacy-preserving, interpretable, and zero-shot fine-tuned table-based arithmetic reasoning in fully localized settings.
📝 Abstract
Recent advancements in artificial intelligence have sparked interest in industrial agents capable of supporting analysts in regulated sectors, such as finance and healthcare, within tabular data workflows. A key capability for such systems is performing accurate arithmetic operations on structured data while ensuring sensitive information never leaves secure, on-premises environments. Here, we introduce an error-driven optimization framework for arithmetic reasoning that enhances a Code Generation Agent (CGA), specifically applied to on-premises small language models (SLMs). Through a systematic evaluation of a leading SLM (Qwen3 4B), we find that while the base model exhibits fundamental limitations in arithmetic tasks, our proposed error-driven method, which clusters erroneous predictions to refine prompt-rules iteratively, dramatically improves performance, elevating the model's accuracy to 70.8%. Our results suggest that developing reliable, interpretable, and industrially deployable AI assistants can be achieved not only through costly fine-tuning but also via systematic, error-driven prompt optimization, enabling small models to surpass larger language models (GPT-3.5 Turbo) in a privacy-compliant manner.