LATTICE: Evaluating Decision Support Utility of Crypto Agents

2026-04-28

AI Summary

Existing evaluations of cryptocurrency agents predominantly emphasize reasoning capabilities or outcome-based metrics, overlooking a systematic assessment of their ability to support user decision-making. This work proposes LATTICE, the first benchmark framework specifically designed to evaluate user decision support in crypto assistants, encompassing six dimensions and 16 task categories that span end-to-end agent workflows. Built upon large language models, LATTICE enables automatic scoring without requiring expert annotations or external data, while supporting dynamic expansion of both dimensions and tasks. The framework is applied to evaluate six leading crypto copilots across 1,200 real-world user queries. Results reveal that despite similar overall scores, the agents exhibit substantial performance disparities across specific dimensions and tasks, highlighting the importance of aligning agent selection with particular user needs.

Abstract

We introduce LATTICE, a benchmark for evaluating the decision support utility of crypto agents in realistic user-facing scenarios. Prior crypto agent benchmarks mainly focus on reasoning-based or outcome-based evaluation, but do not assess agents' ability to assist user decision-making. LATTICE addresses this gap by: (1) defining six evaluation dimensions that capture key decision support properties; (2) proposing 16 task types that span the end-to-end crypto copilot workflow; and (3) using LLM judges to automatically score agent outputs based on these dimensions and tasks. Crucially, the dimensions and tasks are designed to be evaluable at scale using LLM judges, without relying on ground truth from expert annotators or external data sources. In lieu of these dependencies, LATTICE's LLM judge rubrics can be continually audited and updated given new dimensions, tasks, criteria, and human feedback, thus promoting reliable and extensible evaluation. While other benchmarks often compare foundation models sharing a generic agent framework, we use LATTICE to assess production-level agents used in actual crypto copilot products, reflecting the importance of orchestration and UI/UX design in determining agent quality. In this paper, we evaluate six real-world crypto copilots on 1,200 diverse queries and report breakdowns across dimensions, tasks, and query categories. Our experiments show that most of the tested copilots achieve comparable aggregate scores, but differ more significantly on dimension-level and task-level performance. This pattern suggests meaningful trade-offs in decision support quality: users with different priorities may be better served by different copilots than the aggregate rankings alone would indicate. To support reproducible research, we open-source all LATTICE code and data used in this paper.

Problem

• crypto agents
• decision support
• benchmark
• evaluation
• user-facing scenarios

Innovation

• decision support evaluation
• crypto agents
• LLM-based judging
• benchmark design
• end-to-end task workflow