Litmus: Zero-Label, Code-Driven Metric Specification for Evaluating AI Systems

2026-06-22

AI Summary

Current evaluations of AI systems often suffer from ambiguous objectives, poorly justified metric selection, and limited verifiability, particularly lacking interpretable and comprehensive assessment frameworks for complex, multi-stage pipelines. This work proposes Litmus, the first approach to automatically generate evaluation specifications without human-provided labels. By integrating program analysis with natural language interaction, Litmus extracts evaluation intent directly from source code and targeted queries, then employs constraint-based reasoning to dynamically construct stage-specific metric suites aligned with each pipeline component’s objective—shifting the focus from “how to compute” to “what to measure.” Experiments on three real-world AI pipelines demonstrate that Litmus significantly outperforms baseline methods in terms of focal-point coverage, stage-wise breadth, and metric validity, achieving a Spearman correlation of 0.72 on scientific question-answering tasks.

Abstract

As agentic LLM systems move from prototypes to deployment across increasingly diverse domains, evaluating them has become both more important and more difficult. The challenge is not only that individual metrics may be unreliable, but that evaluation goals are often left implicit. Without a clear account of what a system is expected to do, how it can fail, and which failures matter, metric choices become difficult to justify, interpret, or validate. We present Litmus, a zero-label system that designs evaluation and monitoring metrics for AI pipelines by eliciting evaluation intent from source code and targeted interrogation. Instead of assuming that the evaluation target is already known, Litmus first identifies what must be measured and why, then converts those answers into constraints for constructing a justified, per-stage metric portfolio. We evaluate Litmus on three real, code-defined AI pipelines - financial account grouping, scientific QA, and inherent risk assessment - against AutoMetrics and three DynamicRubric baselines. Litmus achieves the broadest or tied-broadest concern coverage, spans more pipeline stages, produces a near-zero-redundancy portfolio, and ranks first in validity against per-row quality labels on all three pipelines - decisively on scientific QA (Spearman $ρ=0.72$ vs. less than $0.47$ for every baseline), and within overlapping confidence intervals in relation to two components of the audit framework despite using no labels during metric design. Our results support a shift from automatic metric implementation to automatic metric specification: before asking which metric to compute, evaluation systems should ask what must be measured and why.

Problem

• AI evaluation
• metric specification
• evaluation intent
• zero-label
• agentic LLM systems

Innovation

• zero-label evaluation
• code-driven metrics
• evaluation intent
• metric specification
• AI pipeline monitoring