Ranked Voting based Self-Consistency of Large Language Models

2025-05-16

AI Summary

To address the bias in self-consistency evaluation caused by single-answer sampling in chain-of-thought (CoT) reasoning for large language models, this paper proposes Rank-CoT, a ranking-enhanced self-consistency framework. Rank-CoT generates multiple candidate answers per reasoning path, each explicitly ranked, and introduces a ranking-aware aggregation mechanism—comprising immediate-elimination voting, Borda counting, and mean reciprocal rank—to jointly leverage cross-path confidence scores. This work is the first to systematically integrate answer ranking generation and ranked voting into the self-consistency paradigm, substantially improving reasoning reliability and robustness. Rank-CoT consistently outperforms state-of-the-art self-consistency baselines across three multiple-choice and three open-ended QA benchmarks. It is compatible with both open-source and proprietary large language models. The implementation is publicly available.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
• Reasoning under Uncertainty: Causality

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Majority voting is considered an effective method to enhance chain-of-thought reasoning, as it selects the answer with the highest"self-consistency"among different reasoning paths (Wang et al., 2023). However, previous chain-of-thought reasoning methods typically generate only a single answer in each trial, thereby ignoring the possibility of other potential answers. As a result, these alternative answers are often overlooked in subsequent voting processes. In this work, we propose to generate ranked answers in each reasoning process and conduct ranked voting among multiple ranked answers from different responses, thereby making the overall self-consistency more reliable. Specifically, we use three ranked voting methods: Instant-runoff voting, Borda count voting, and mean reciprocal rank voting. We validate our methods on six datasets, including three multiple-choice and three open-ended question-answering tasks, using both advanced open-source and closed-source large language models. Extensive experimental results indicate that our proposed method outperforms the baselines, showcasing the potential of leveraging the information of ranked answers and using ranked voting to improve reasoning performance. The code is available at https://github.com/szu-tera/RankedVotingSC.

Problem

• Improves self-consistency in LLM reasoning via ranked voting
• Addresses overlooked potential answers in majority voting methods
• Validates ranked voting on multiple QA tasks and models

Innovation

• Generates ranked answers in reasoning processes
• Uses ranked voting methods for self-consistency
• Validates on multiple datasets and models