RubricsTree: Scalable and Evolving Open-Ended Evaluation of Personal Health Agents across Health Memory and Medical Skills

2026-06-16

AI Summary

This work addresses the open-ended evaluation bottleneck hindering large-scale deployment of personal health agents, where physician annotations are reliable yet unscalable, and large language model–based automatic evaluations suffer from subjectivity and insufficient clinical alignment. To overcome this, the authors propose RubricsTree—the first expert-aligned, auditable, and continuously evolving evaluation framework. Through an iterative human-AI collaborative process led by senior physicians, the framework distills over 100 atomic, clinically verifiable Boolean rules from 4,000 real user queries, organizing them into a hierarchical rubric structure complemented by a context-aware adaptive routing mechanism that dynamically activates relevant rule subsets. RubricsTree significantly outperforms strong baselines in expert alignment and effectively mitigates context-degraded responses. When employed as instructions, feedback, or reward signals for model optimization, it yields up to a 66% relative performance gain on HealthBench across Gemini, GPT, and Qwen model families.

Abstract

The LLM-empowered personal health agents with user health (sensor) metrics have offered a promising pathway to alleviate global disparities in healthcare access. However, large-scale clinical deployment remains constrained by an open-ended evaluation bottleneck: physician annotation is reliable but costly and unscalable, while LLM-as-a-judge evaluators are scalable but subjective, inconsistent, and sometimes clinically misaligned. We introduce RubricsTree, a scalable evaluation framework with an expert-aligned hierarchical taxonomy of over 100 atomic, clinically-verifiable Boolean rubrics, evolving from the insights of 4,000 real user queries through an iterative human-in-the-loop curation protocol with an expertise panel led by an experienced physician. A context-aware adaptive router activates only the relevant auto-weighted rubric subset per query, providing the throughput needed for scalable evaluation with expert-aligned quality. Through a systematic meta-evaluation, we show that RubricsTree (i) substantially exceeds a strong large-scale evaluation baseline in expert alignment on challenging open-ended queries; (ii) reliably penalizes contextually degraded responses; and (iii) when used as structured instructions, text feedback, or training rewards for performance optimization, yields up to ~66% relative gains on HealthBench for Gemini, GPT, and Qwen model families. RubricsTree thus provides a scalable, auditable, and evolving evaluation infrastructure required for the continuous optimization of product-level personal healthcare AI.

Problem

• personal health agents
• open-ended evaluation
• clinical alignment
• scalable assessment
• healthcare AI

Innovation

• RubricsTree
• open-ended evaluation
• clinically-verifiable rubrics
• adaptive routing
• human-in-the-loop curation