Selected work

Representative Papers

RM -RF: Reward Model for Run-Free Unit Test Evaluation

Jan 19, 2026

This work proposes the first lightweight reward model capable of predicting multidimensional test quality—including executability, code coverage, and mutation kill rate—without executing tests, thereby circumventing the high latency and resource overhead inherent in traditional unit test evaluation that relies on repeated compilation and execution. Leveraging a multilingual dataset constructed from Java, Python, and Go, the model is trained using parameter-efficient strategies such as zero-shot inference, full fine-tuning, and LoRA. It achieves an average F1 score of 0.69 across all three quality metrics. By eliminating the need for actual test runs, this approach substantially reduces evaluation cost and latency, offering an efficient foundation for large-scale test generation and reinforcement learning–based test optimization.

1 citationsRead paper

Little Brains, Big Feats: Exploring Compact Language Models

Jun 29, 2026

This study addresses the lack of systematic evaluation of small language models (SLMs) in retrieval-augmented generation (RAG) systems, particularly their potential for deployment on resource-constrained devices. It presents the first comprehensive assessment of SLMs in the RAG generation phase, benchmarking performance across diverse domains using both open-source and proprietary datasets. The work further demonstrates end-side inference entirely on CPU-based hardware without GPU acceleration. Experimental results show that SLMs can operate efficiently under such constraints, substantially reducing computational overhead while maintaining reasonable response times and generating high-quality outputs. These findings establish a viable pathway for deploying lightweight, edge-compatible AI systems leveraging RAG architectures.

0 citationsRead paper

asr_eval: Algorithms and tools for multi-reference and streaming speech recognition evaluation

Jan 28, 2026

This work addresses key challenges in speech recognition evaluation under scenarios involving multiple reference transcriptions, non-Latin scripts, long-form audio, and streaming conditions. The authors propose an enhanced string alignment algorithm that accommodates multiple references and arbitrary-length insertions, enabling more accurate assessment of automatic speech recognition (ASR) systems. They construct DiverseSpeech-Ru, a high-quality Russian long-form multi-reference test set, and develop a unified evaluation and visualization toolkit supporting both streaming and offline ASR paradigms. Their approach not only uncovers model overfitting to specific transcription styles but also substantially improves the accuracy and interpretability of evaluations for multi-reference and streaming ASR systems.

0 citationsRead paper

Pisets: A Robust Speech Recognition System for Lectures and Interviews

Jan 26, 2026North American Chapter of the Association for Computational Linguistics

This work addresses the susceptibility of Whisper to transcription errors and hallucinations in long-form audio scenarios such as lectures and interviews. To mitigate these issues, the authors propose a three-stage cascaded architecture: an initial transcription is generated using Wav2Vec2, followed by false-positive filtering via an Audio Spectrogram Transformer (AST), and finally refined output production by Whisper. The approach integrates uncertainty modeling and curriculum learning, and is trained on a diverse multilingual Russian speech corpus. Experimental results demonstrate that the proposed system significantly outperforms both Whisper and WhisperX across varying acoustic conditions, yielding substantial improvements in accuracy and robustness for long-form audio transcription. The implementation has been made publicly available.

0 citationsRead paper

TAM-Eval: Evaluating LLMs for Automated Unit Test Maintenance

Jan 26, 2026

This work addresses the limited applicability of current large language models (LLMs) in unit testing, which typically focus on isolated test generation or assertion prediction and struggle to support holistic test suite maintenance. We propose TAM-Eval, a novel framework and benchmark that, for the first time, constructs a multilingual dataset of 1,539 real-world test maintenance scenarios at the test-file level across Python, Java, and Go, contextualized within actual software repositories. TAM-Eval introduces a reference-free evaluation protocol to uniformly assess both base models and agent-based workflows. Comprehensive evaluation using test pass rate, code coverage, and mutation score reveals that state-of-the-art models exhibit significant limitations in authentic tasks involving test creation, repair, and updating, thereby highlighting the substantial challenges and research opportunities in this domain.

0 citationsRead paper

Recent publications

Latest Papers

Little Brains, Big Feats: Exploring Compact Language Models

Jun 29, 2026

This study addresses the lack of systematic evaluation of small language models (SLMs) in retrieval-augmented generation (RAG) systems, particularly their potential for deployment on resource-constrained devices. It presents the first comprehensive assessment of SLMs in the RAG generation phase, benchmarking performance across diverse domains using both open-source and proprietary datasets. The work further demonstrates end-side inference entirely on CPU-based hardware without GPU acceleration. Experimental results show that SLMs can operate efficiently under such constraints, substantially reducing computational overhead while maintaining reasonable response times and generating high-quality outputs. These findings establish a viable pathway for deploying lightweight, edge-compatible AI systems leveraging RAG architectures.

0 citationsRead paper

asr_eval: Algorithms and tools for multi-reference and streaming speech recognition evaluation

Jan 28, 2026

This work addresses key challenges in speech recognition evaluation under scenarios involving multiple reference transcriptions, non-Latin scripts, long-form audio, and streaming conditions. The authors propose an enhanced string alignment algorithm that accommodates multiple references and arbitrary-length insertions, enabling more accurate assessment of automatic speech recognition (ASR) systems. They construct DiverseSpeech-Ru, a high-quality Russian long-form multi-reference test set, and develop a unified evaluation and visualization toolkit supporting both streaming and offline ASR paradigms. Their approach not only uncovers model overfitting to specific transcription styles but also substantially improves the accuracy and interpretability of evaluations for multi-reference and streaming ASR systems.

0 citationsRead paper

Pisets: A Robust Speech Recognition System for Lectures and Interviews

Jan 26, 2026North American Chapter of the Association for Computational Linguistics

This work addresses the susceptibility of Whisper to transcription errors and hallucinations in long-form audio scenarios such as lectures and interviews. To mitigate these issues, the authors propose a three-stage cascaded architecture: an initial transcription is generated using Wav2Vec2, followed by false-positive filtering via an Audio Spectrogram Transformer (AST), and finally refined output production by Whisper. The approach integrates uncertainty modeling and curriculum learning, and is trained on a diverse multilingual Russian speech corpus. Experimental results demonstrate that the proposed system significantly outperforms both Whisper and WhisperX across varying acoustic conditions, yielding substantial improvements in accuracy and robustness for long-form audio transcription. The implementation has been made publicly available.

0 citationsRead paper

TAM-Eval: Evaluating LLMs for Automated Unit Test Maintenance

Jan 26, 2026

This work addresses the limited applicability of current large language models (LLMs) in unit testing, which typically focus on isolated test generation or assertion prediction and struggle to support holistic test suite maintenance. We propose TAM-Eval, a novel framework and benchmark that, for the first time, constructs a multilingual dataset of 1,539 real-world test maintenance scenarios at the test-file level across Python, Java, and Go, contextualized within actual software repositories. TAM-Eval introduces a reference-free evaluation protocol to uniformly assess both base models and agent-based workflows. Comprehensive evaluation using test pass rate, code coverage, and mutation score reveals that state-of-the-art models exhibit significant limitations in authentic tasks involving test creation, repair, and updating, thereby highlighting the substantial challenges and research opportunities in this domain.

0 citationsRead paper

RM -RF: Reward Model for Run-Free Unit Test Evaluation

Jan 19, 2026

This work proposes the first lightweight reward model capable of predicting multidimensional test quality—including executability, code coverage, and mutation kill rate—without executing tests, thereby circumventing the high latency and resource overhead inherent in traditional unit test evaluation that relies on repeated compilation and execution. Leveraging a multilingual dataset constructed from Java, Python, and Go, the model is trained using parameter-efficient strategies such as zero-shot inference, full fine-tuning, and LoRA. It achieves an average F1 score of 0.69 across all three quality metrics. By eliminating the need for actual test runs, this approach substantially reduces evaluation cost and latency, offering an efficient foundation for large-scale test generation and reinforcement learning–based test optimization.

1 citationsRead paper