CAFE: A Compound-AI Factorial Evaluation Framework

2026-07-11

AI Summary

This work addresses the challenge of quantifying the impact of interchangeable components in composite AI systems on output quality and the lack of effective methods for identifying critical factors and their interactions. The study introduces factorial experimental design to this domain, proposing a configurable evaluation framework that treats system components as experimental factors. By running multi-configuration experiments and combining large language model outputs with human ratings, the framework employs mixed-effects models to disentangle individual component contributions, interaction effects, and statistical significance. Experiments on HotpotQA demonstrate that the approach accurately recovers ground-truth factor effects, maintains calibration, and enables comprehensive analysis—including effect sizes, significance levels, cost–latency trade-offs, and rater consistency—while supporting both optimal configuration search and interpretable attribution.

Abstract

We introduce CAFE (Compound-AI Factorial Evaluation), an open-source platform that brings design of experiments to the evaluation of compound AI systems (CAIS). Such systems expose many interchangeable choices - e.g. which retriever, model, or prompt - and practitioners rarely know which of them most affects answer quality. With CAFE, a practitioner registers each swappable component of a pipeline as a factor to build a factorial design over the chosen factors, run the resulting configurations, and score the answers on a shared rubric using a configurable LLM judge together with human raters. From these ratings it attributes answer-quality variance to the components and their interactions with mixed-effects models and reports effect sizes, significance, the best configuration, cost and latency trade-offs, and judge-human reliability. Whereas existing tools mostly either search for a good configuration or score outputs in isolation, CAFE also explains which component drives quality and whether an observed difference is significant. We validate CAFE on a retrieval-augmented question-answering (QA) pipeline over the HotpotQA benchmark dataset, where it recovers planted factor effects and stays calibrated under a permutation null. CAFE is released as a Python package and as a Web application.

Problem

• compound AI systems
• factorial evaluation
• answer quality
• component interaction
• design of experiments

Innovation

• Factorial Design
• Compound AI Systems
• Mixed-Effects Models
• LLM-based Evaluation
• Causal Attribution