Who's Keeping Score? Interactive Steering of LLM-Powered Scoring with Attune

2026-08-14

AI Summary

This study addresses the limitations of large language models in text scoring, specifically their lack of global comprehension and local consistency. We propose Attune, a novel hybrid proactive framework that establishes global cognition through pairwise comparisons and induces editable rules via bottom-up generalization. By integrating constraint compilation, the system enables deterministic interactions and supports user-driven logic corrections to guide rescoring, thereby facilitating controllable human-AI collaborative evaluation. Extensive experiments across three tasks and multi-domain expert studies validate the effectiveness of this approach. Results demonstrate significant improvements in scoring accuracy, consistency, and interpretability, establishing Attune as a new paradigm for controllable text assessment.

Abstract

Large language models (LLMs) are increasingly used to score text records at scale (e.g., rating candidate resumes on a 1-5 scale). However, existing LLM-powered approaches do not account for the fact that effective scoring requires both holistic understanding of records and locally consistent judgments across similar ones. We present Attune, a mixed-initiative system for steerable LLM-powered scoring. Given a task description and scoring range, Attune performs pairwise comparisons across records to develop a global understanding first, and then resolves these comparisons into consistent score assignments-deriving scoring criteria and rules bottom-up in the process. These serve as shared representations of scoring logic that users can inspect and edit. Based on insights from a formative study (n = 12), Attune's interface introduces novel steering interactions that allow users to deterministically refine scoring logic. Users can provide examples, directly edit criteria, rules, or target distributions, and give natural language feedback-with all refinements compiling into constraints that guide re-scoring. We validate our approach through a technical evaluation across three workloads and a user study with domain experts (n = 8) in healthcare, law, education, and AI evaluation.

Problem

• LLM-powered scoring
• scoring consistency
• interactive steering
• holistic understanding

Innovation

• Steerable LLM Scoring
• Mixed-Initiative Interaction
• Pairwise Comparison
• Bottom-Up Criteria Derivation
• Deterministic Refinement