Stratified Consistency Distillation for Natural Language Formalization

2026-08-31

AI Summary

为提高自然语言到逻辑公式的翻译准确性，提出了一种基于微调的分层一致性蒸馏方法，通过生成多个逻辑翻译并聚类选择伪标签来改进模型。

Abstract

Neurosymbolic reasoning has shown promising success in addressing complex reasoning tasks by combining large language models (LLMs) and symbolic solvers. While this approach shows promise, a fundamental challenge remains: improving the accuracy of translations from natural language to logical formulas. Current methods predominantly rely on prompt engineering, which is difficult to scale across different domains and input formats. Drawing inspiration from the success of fine-tuning in other model adaptation and alignment applications, we propose a fine-tuning-based Stratified Consistency Distillation approach: (1) We generate K logical translations per input using a frontier LLM and cluster them by semantic equivalence (2) Based on the entropy level, we apply majority voting (low entropy), LLM-as-a-Judge (medium entropy), or unification/abstention (high entropy), and (3) fine-tune a smaller model using the selected pseudo-labels. Our experiments show significant and consistent improvements in both Pass@K and our novel Equivalent Logical Similarity metrics, demonstrating the potential of advancing logical translation through consistency distillation.

Problem

• Natural Language Formalization
• Neurosymbolic Reasoning
• Logical Translation

Innovation

• Stratified Consistency Distillation
• fine-tuning
• majority voting
• LLM-as-a-Judge
• unification/abstention