StepFun-Formalizer: Unlocking the Autoformalization Potential of LLMs through Knowledge-Reasoning Fusion

2025-08-06

AI Summary

Current large language models exhibit low accuracy in automatic formalization of natural-language mathematical statements, primarily due to the lack of joint acquisition of formal-language domain knowledge and the ability to map informal natural language to formal logical representations. This paper introduces ThinkingF, the first systematic training framework that jointly integrates these two capabilities: (1) a formal-knowledge-augmented dataset, (2) expert-guided, templated reasoning trajectory generation, and (3) a hybrid optimization strategy combining knowledge distillation, supervised fine-tuning (SFT), and reinforcement learning with verifiable reward (RLVR). Evaluated on 7B and 32B models, StepFun-Formalizer-32B achieves BEq@1 scores of 40.5% on FormalMATH-Lite and 26.7% on ProverBench—substantially outperforming all general-purpose and task-specific baselines and establishing new state-of-the-art performance.

Technology Category

• Knowledge Representation and Reasoning: Automated Reasoning and Theorem Proving
• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Autoformalization aims to translate natural-language mathematical statements into a formal language. While LLMs have accelerated progress in this area, existing methods still suffer from low accuracy. We identify two key abilities for effective autoformalization: comprehensive mastery of formal-language domain knowledge, and reasoning capability of natural language problem understanding and informal-formal alignment. Without the former, a model cannot identify the correct formal objects; without the latter, it struggles to interpret real-world contexts and map them precisely into formal expressions. To address these gaps, we introduce ThinkingF, a data synthesis and training pipeline that improves both abilities. First, we construct two datasets: one by distilling and selecting large-scale examples rich in formal knowledge, and another by generating informal-to-formal reasoning trajectories guided by expert-designed templates. We then apply SFT and RLVR with these datasets to further fuse and refine the two abilities. The resulting 7B and 32B models exhibit both comprehensive formal knowledge and strong informal-to-formal reasoning. Notably, StepFun-Formalizer-32B achieves SOTA BEq@1 scores of 40.5% on FormalMATH-Lite and 26.7% on ProverBench, surpassing all prior general-purpose and specialized models.

Problem

• Improving autoformalization accuracy in LLMs
• Enhancing formal-language knowledge and reasoning capabilities
• Bridging natural-language understanding to formal expression mapping

Innovation

• Knowledge-Reasoning Fusion for autoformalization
• Data synthesis with expert-guided reasoning trajectories
• SFT and RLVR training to refine model abilities