Selected work

Representative Papers

Process Reward Models Meet Planning: Generating Precise and Scalable Datasets for Step-Level Rewards

Apr 20, 2026

This work addresses the limitations of existing process reward models (PRMs), which rely on costly and error-prone human annotations and are largely confined to mathematical domains, thereby lacking the fine-grained feedback required for general reasoning tasks. To overcome these challenges, the study introduces a novel paradigm that integrates automated planning with PRM training. Specifically, logical problems are formalized using the Planning Domain Definition Language (PDDL), and large-scale datasets comprising millions of reasoning steps are generated via automated planning algorithms. This approach yields a cross-domain, scalable framework for PRM training. Experimental results demonstrate significant performance gains across multiple mathematical and non-mathematical reasoning benchmarks, confirming the effectiveness and generalizability of planning-generated data in enhancing model reasoning capabilities.

0 citationsRead paper

LiteraryQA: Towards Effective Evaluation of Long-document Narrative QA

Oct 15, 2025

Existing narrative long-document QA benchmarks (e.g., NarrativeQA) suffer from high document noise and low-quality question-answer pairs, undermining evaluation reliability. Method: We introduce LiteraryQA—a rigorously curated high-quality subset—built via a human–LLM collaborative verification pipeline for data cleaning. We systematically evaluate automated metrics, analyzing their correlation with human judgments across diverse long-context LLMs. Contribution/Results: We find that conventional n-gram metrics exhibit weak correlation with human assessments, whereas lightweight open-source LLMs (e.g., Qwen2-7B) used as judges achieve system-level ranking consistency with human annotators. We publicly release the filtered dataset and an integrated evaluation framework, and conduct comprehensive benchmarking across multiple long-context LLMs. This work provides the first empirical validation of LLM-as-a-Judge for narrative QA evaluation—demonstrating both its effectiveness and computational efficiency.

0 citationsRead paper

Do Large Language Models Understand Word Senses?

Sep 17, 2025

This study addresses the foundational question of whether large language models (LLMs) genuinely comprehend context-dependent word semantics. We systematically design four evaluation tasks—word sense disambiguation (WSD), definition generation, free-form explanation, and example sentence generation—spanning multiple domains and difficulty levels. For the first time, we comparatively assess instruction-tuned LLMs (GPT-4o, DeepSeek-V3) against dedicated WSD systems. Results show that LLMs match state-of-the-art specialized models on WSD, while achieving 98% accuracy on generative semantic understanding tasks—substantially outperforming baselines. Our key contribution is the construction of the first integrated evaluation framework unifying discriminative and generative paradigms for lexical semantic understanding. Empirical findings demonstrate that current mainstream LLMs possess fine-grained, context-sensitive word sense comprehension approaching human-level proficiency.

0 citationsRead paper

Optimizing LLMs for Italian: Reducing Token Fertility and Enhancing Efficiency Through Vocabulary Adaptation

Apr 23, 2025

English-dominant large language models exhibit high token fertility and suboptimal inference efficiency on Italian. To address this, we propose Semantic-Aligned Vocabulary Adaptation (SAVA), the first method to jointly enforce semantic consistency constraints and differentiable vocabulary replacement for cross-lingual vocabulary optimization. SAVA employs neural mapping to achieve efficient monolingual adaptation without compromising multi-task generalization. Experiments demonstrate that SAVA reduces token fertility by 25% for Mistral-7B on Italian, compresses the Llama-3.1-8B vocabulary—eliminating approximately one billion parameters—and restores downstream performance with minimal Italian-language continual training. The adapted models retain competitiveness on multiple-choice and generative tasks. This work establishes a novel paradigm for efficient LLM adaptation to low-resource languages.

0 citationsRead paper

Right Answer, Wrong Score: Uncovering the Inconsistencies of LLM Evaluation in Multiple-Choice Question Answering

Mar 19, 2025

This paper identifies systematic inconsistencies in evaluating large language models (LLMs) on multiple-choice question answering (MCQA): conventional answer extraction methods—e.g., keyword matching—diverge significantly from human judgments when LLMs first generate free-text reasoning before selecting an answer, and exhibit strong sensitivity to prompt formatting. Method: Through controlled experiments across diverse MCQA benchmarks, human-annotated calibration, and LLM-driven consistency analysis, the study rigorously isolates the interplay between prompt format constraints and free-form reasoning capability. Contribution/Results: It establishes, for the first time, a fundamental trade-off between format adherence and reasoning flexibility; demonstrates that standard evaluation underestimates true model competence; and reveals systematic biases in LLM-based self-extraction. Collectively, findings expose severe distortion in current MCQA evaluation practices, underscoring an urgent need for standardization. The work provides empirical evidence and methodological guidance for developing robust, consistent MCQA evaluation frameworks.

0 citationsRead paper

Recent publications

Latest Papers

Process Reward Models Meet Planning: Generating Precise and Scalable Datasets for Step-Level Rewards

Apr 20, 2026

This work addresses the limitations of existing process reward models (PRMs), which rely on costly and error-prone human annotations and are largely confined to mathematical domains, thereby lacking the fine-grained feedback required for general reasoning tasks. To overcome these challenges, the study introduces a novel paradigm that integrates automated planning with PRM training. Specifically, logical problems are formalized using the Planning Domain Definition Language (PDDL), and large-scale datasets comprising millions of reasoning steps are generated via automated planning algorithms. This approach yields a cross-domain, scalable framework for PRM training. Experimental results demonstrate significant performance gains across multiple mathematical and non-mathematical reasoning benchmarks, confirming the effectiveness and generalizability of planning-generated data in enhancing model reasoning capabilities.

0 citationsRead paper

LiteraryQA: Towards Effective Evaluation of Long-document Narrative QA

Oct 15, 2025

Existing narrative long-document QA benchmarks (e.g., NarrativeQA) suffer from high document noise and low-quality question-answer pairs, undermining evaluation reliability. Method: We introduce LiteraryQA—a rigorously curated high-quality subset—built via a human–LLM collaborative verification pipeline for data cleaning. We systematically evaluate automated metrics, analyzing their correlation with human judgments across diverse long-context LLMs. Contribution/Results: We find that conventional n-gram metrics exhibit weak correlation with human assessments, whereas lightweight open-source LLMs (e.g., Qwen2-7B) used as judges achieve system-level ranking consistency with human annotators. We publicly release the filtered dataset and an integrated evaluation framework, and conduct comprehensive benchmarking across multiple long-context LLMs. This work provides the first empirical validation of LLM-as-a-Judge for narrative QA evaluation—demonstrating both its effectiveness and computational efficiency.

0 citationsRead paper

Do Large Language Models Understand Word Senses?

Sep 17, 2025

This study addresses the foundational question of whether large language models (LLMs) genuinely comprehend context-dependent word semantics. We systematically design four evaluation tasks—word sense disambiguation (WSD), definition generation, free-form explanation, and example sentence generation—spanning multiple domains and difficulty levels. For the first time, we comparatively assess instruction-tuned LLMs (GPT-4o, DeepSeek-V3) against dedicated WSD systems. Results show that LLMs match state-of-the-art specialized models on WSD, while achieving 98% accuracy on generative semantic understanding tasks—substantially outperforming baselines. Our key contribution is the construction of the first integrated evaluation framework unifying discriminative and generative paradigms for lexical semantic understanding. Empirical findings demonstrate that current mainstream LLMs possess fine-grained, context-sensitive word sense comprehension approaching human-level proficiency.

0 citationsRead paper

Optimizing LLMs for Italian: Reducing Token Fertility and Enhancing Efficiency Through Vocabulary Adaptation

Apr 23, 2025

English-dominant large language models exhibit high token fertility and suboptimal inference efficiency on Italian. To address this, we propose Semantic-Aligned Vocabulary Adaptation (SAVA), the first method to jointly enforce semantic consistency constraints and differentiable vocabulary replacement for cross-lingual vocabulary optimization. SAVA employs neural mapping to achieve efficient monolingual adaptation without compromising multi-task generalization. Experiments demonstrate that SAVA reduces token fertility by 25% for Mistral-7B on Italian, compresses the Llama-3.1-8B vocabulary—eliminating approximately one billion parameters—and restores downstream performance with minimal Italian-language continual training. The adapted models retain competitiveness on multiple-choice and generative tasks. This work establishes a novel paradigm for efficient LLM adaptation to low-resource languages.

0 citationsRead paper

Right Answer, Wrong Score: Uncovering the Inconsistencies of LLM Evaluation in Multiple-Choice Question Answering

Mar 19, 2025

This paper identifies systematic inconsistencies in evaluating large language models (LLMs) on multiple-choice question answering (MCQA): conventional answer extraction methods—e.g., keyword matching—diverge significantly from human judgments when LLMs first generate free-text reasoning before selecting an answer, and exhibit strong sensitivity to prompt formatting. Method: Through controlled experiments across diverse MCQA benchmarks, human-annotated calibration, and LLM-driven consistency analysis, the study rigorously isolates the interplay between prompt format constraints and free-form reasoning capability. Contribution/Results: It establishes, for the first time, a fundamental trade-off between format adherence and reasoning flexibility; demonstrates that standard evaluation underestimates true model competence; and reveals systematic biases in LLM-based self-extraction. Collectively, findings expose severe distortion in current MCQA evaluation practices, underscoring an urgent need for standardization. The work provides empirical evidence and methodological guidance for developing robust, consistent MCQA evaluation frameworks.

0 citationsRead paper