RCScore: Quantifying Response Consistency in Large Language Models

2025-10-30

AI Summary

Existing LLM evaluations predominantly rely on single instruction templates, overlooking instruction-style sensitivity—a critical factor for real-world deployment. Method: We propose RCScore, the first multidimensional evaluation framework that quantifies how instruction phrasing affects model responses. It introduces Cross-Response Similarity (CRS) to measure stylistic consistency across diverse instruction variants, systematically transforming instruction styles for benchmark tasks and computing CRS via multiple similarity dimensions. Contribution/Results: Experiments across four reasoning benchmarks and ten mainstream LLMs reveal instruction-style variations induce up to 16.7 percentage points in accuracy fluctuation. CRS strongly correlates with task accuracy and effectively characterizes how decoding strategies and model scale influence stylistic robustness—capturing performance dynamics invisible to conventional accuracy metrics.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Sentiment Analysis, Stylistic Analysis, and Argument Mining
• Cognitive Modeling & Cognitive Systems: Computational Creativity

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success

Abstract

Current LLM evaluations often rely on a single instruction template, overlooking models' sensitivity to instruction style-a critical aspect for real-world deployments. We present RCScore, a multi-dimensional framework quantifying how instruction formulation affects model responses. By systematically transforming benchmark problems into multiple instruction styles, RCScore reveals performance variations undetected by conventional metrics. Our experiments across ten LLMs on four reasoning benchmarks demonstrate that instruction style can shift accuracy by up to 16.7% points. We introduce Cross-Response Similarity (CRS), a method applying RCScore metrics to measure stylistic self-consistency, and establish its strong correlation with task accuracy, suggesting consistency as a valuable proxy for model reliability. Additional findings show that deterministic decoding produces more stylistically stable outputs, and model scale correlates positively with cross-style consistency. RCScore offers a principled approach to assess instruction robustness.

Problem

• Quantifying LLM sensitivity to instruction style variations
• Measuring performance fluctuations across different instruction formulations
• Establishing consistency as a proxy for model reliability

Innovation

• RCScore quantifies instruction style impact on responses
• Introduces Cross-Response Similarity for stylistic self-consistency
• Systematically transforms benchmarks into multiple instruction styles