DeepInsight: A Unified Evaluation Infrastructure Across the Physical AI Stack

2026-06-16

AI Summary

Existing evaluation frameworks struggle to uniformly cover the full stack of physical AI systems, whose scales span over three orders of magnitude and exhibit diverse modalities and resource requirements, thereby hindering the diagnosis of cross-layer regressions. This work proposes DeepInsight—a unified evaluation infrastructure that enables end-to-end co-evaluation of heterogeneous components within a single runtime. By introducing three core abstractions—tasks, resources, and results—DeepInsight integrates a unified episode-driven execution model, standardized resource handle protocols, and global trace identifiers. The system has been deployed on a three-tier humanoid robot architecture, successfully reproducing results from mainstream models while achieving single-node acceleration and near-linear multi-node scalability. Notably, it enables, for the first time, precise identification of the root causes behind cross-layer performance regressions.

Abstract

Evaluating a Physical AI stack spans operators that differ by more than three orders of magnitude -- from a single foundation-model decoding step to thousands of physics ticks of whole-body control -- varying orthogonally in modality, reward semantics, and resource profile. No existing framework spans this range, so the stack is evaluated today by stitching together separate harnesses that share neither runtime nor scoring, preserving each segment's local validity but losing the shared identity needed to diagnose cross-layer regressions. We present DeepInsight, an evaluation infrastructure that serves this full spectrum on a single runtime. Rather than homogenize the regimes, it preserves their heterogeneity behind three narrow abstractions -- task, resource, and result -- each realized as one invariant shared by every subsystem: one episode driver, one resource-handle protocol implemented by every expensive backend (LLM inference and sandboxed runtimes alike), and one trace identity scheme under which every event is written. Deployed in production across all three layers of an embodied humanoid stack, this single set of invariants onboards new benchmarks largely by configuration. Where mature peer orchestrators exist -- at the foundation-model end -- it reproduces published references and peer-framework readings within their own spread, runs the same suites faster on a single node, and scales near-linearly across nodes. Its distinctive return is diagnostic: because every layer writes into one shared trace, a regression that begins in one layer and surfaces in another stays localizable on that trace -- a cross-layer payoff no federation of per-segment harnesses can reproduce.

Problem

• Physical AI
• evaluation infrastructure
• cross-layer regression
• heterogeneous systems
• unified tracing

Innovation

• unified evaluation infrastructure
• Physical AI Stack
• cross-layer diagnostics
• heterogeneous abstraction
• shared trace identity