OdysseyArena: Benchmarking Large Language Models For Long-Horizon, Active and Inductive Interactions

2026-02-05

AI Summary

Current evaluations of large language models are largely confined to static rules and short-horizon tasks, failing to assess their capacity for long-term autonomous exploration and inductive reasoning about environmental dynamics in complex settings. This work proposes OdysseyArena, a novel evaluation framework that shifts the paradigm from deductive tasks to inductive interaction. By formalizing four classes of interactive primitives, the framework introduces OdysseyArena-Lite—a benchmark comprising 120 tasks—and a more demanding Challenge suite, both supporting interactions exceeding 200 steps. Systematic evaluation of over 15 leading models reveals significant deficiencies in inductive, long-horizon tasks, exposing a critical bottleneck in the development of autonomous agents and offering clear guidance for future research.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Multiagent Systems: Adversarial Agents

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

The rapid advancement of Large Language Models (LLMs) has catalyzed the development of autonomous agents capable of navigating complex environments. However, existing evaluations primarily adopt a deductive paradigm, where agents execute tasks based on explicitly provided rules and static goals, often within limited planning horizons. Crucially, this neglects the inductive necessity for agents to discover latent transition laws from experience autonomously, which is the cornerstone for enabling agentic foresight and sustaining strategic coherence. To bridge this gap, we introduce OdysseyArena, which re-centers agent evaluation on long-horizon, active, and inductive interactions. We formalize and instantiate four primitives, translating abstract transition dynamics into concrete interactive environments. Building upon this, we establish OdysseyArena-Lite for standardized benchmarking, providing a set of 120 tasks to measure an agent's inductive efficiency and long-horizon discovery. Pushing further, we introduce OdysseyArena-Challenge to stress-test agent stability across extreme interaction horizons (e.g.,>200 steps). Extensive experiments on 15+ leading LLMs reveal that even frontier models exhibit a deficiency in inductive scenarios, identifying a critical bottleneck in the pursuit of autonomous discovery in complex environments. Our code and data are available at https://github.com/xufangzhi/Odyssey-Arena

Problem

• inductive reasoning
• long-horizon interaction
• autonomous agents
• transition dynamics
• LLM benchmarking

Innovation

• inductive reasoning
• long-horizon interaction
• autonomous agents
• transition dynamics discovery
• LLM benchmarking