TSNBench: Benchmarking LLM Proficiency in Time-Sensitive Networking

2026-05-10

AI Summary

This work addresses the lack of specialized evaluation benchmarks for large language models (LLMs) in safety-critical communication domains such as Time-Sensitive Networking (TSN) by introducing the first comprehensive TSN benchmark. It comprises 939 expert-validated multiple-choice questions and 100 worst-case delay (WCD) computation tasks, covering Credit-Based Shaper (CBS) and Cyclic Queuing and Forwarding (CQF) mechanisms across diverse topologies and traffic scenarios. The study systematically evaluates 16 prominent LLMs using multiple-choice accuracy, a network-calculus-based exact CBS solver, and closed-form upper bounds for CQF. Results show strong performance on multiple-choice questions (67%–95% accuracy) but substantial errors in WCD prediction—optimal mean absolute percentage errors (MAPE) are 36.2% for CBS and 41.8% for CQF, with most models exceeding 80% error—highlighting critical limitations in LLMs’ numerical reasoning for open-ended tasks and challenging current evaluation paradigms.

Abstract

We present TSNBench, the first benchmark for evaluating large language model (LLM) proficiency in Time-Sensitive Networking (TSN), a suite of IEEE 802.1 standards for deterministic communication with bounded latency in safety-critical domains such as autonomous vehicles, aviation, defense, and industrial automation. While LLMs have been extensively evaluated on general knowledge tasks, their capabilities in safety-critical networking domains remain largely unexplored. TSNBench comprises 939 expert-validated multiple-choice questions (MCQs) covering diverse TSN mechanisms, along with 100 open-ended Worst-Case Delay (WCD) computation tasks for Credit-Based Shaper (CBS) and Cyclic Queuing and Forwarding (CQF) across varying network topologies and traffic conditions. MCQ answers are validated by domain experts, and open-ended ground truth WCD values are computed using a verified Network Calculus (NC) solver for CBS and closed-form mathematical upper bounds for CQF. We evaluate 16 LLMs and find that although models achieve 67 to 95% accuracy on MCQs, they fail substantially on open-ended WCD computation. For CBS, only GPT-5 achieves a Mean Absolute Percentage Error (MAPE) of 36.2%, meaning its predicted WCD deviates by 36.2% of the actual TSN flow delay on average, while most models exceed 80%. For CQF, the best model achieves 41.8% MAPE, with most models clustering between 80% and 100%. Such errors are large relative to TSN latency budgets and can lead to violations of real-time constraints and unsafe configurations. TSNBench demonstrates that MCQ benchmarks may overestimate LLM capabilities in safety-critical networking domains.

Problem

• Time-Sensitive Networking
• Large Language Models
• Worst-Case Delay
• Safety-Critical Systems
• Deterministic Communication

Innovation

• Time-Sensitive Networking
• Large Language Models
• Worst-Case Delay
• Network Calculus
• Benchmarking