🤖 AI Summary
Standard JSON exhibits significant token redundancy when representing tabular data due to repeated key names, which undermines the contextual efficiency of large language models. This work proposes JTON, a JSON-compatible superset format that introduces the novel Zen Grid mechanism to separate column headers from values—encoding values as delimiter-separated sequences—thereby achieving substantial token compression while preserving JSON’s full type system. A high-performance parser implemented in Rust with PyO3 integrates SIMD acceleration and demonstrates, across seven real-world datasets, an average token reduction of 28.5% (up to 60%), improves comprehension accuracy by 0.3 percentage points across ten large language models, achieves 100% syntactically valid output generation for twelve models, and attains a parsing speed 1.4× faster than Python’s native json module.
📝 Abstract
When LLMs process structured data, the serialization format directly affects cost and context utilization. Standard JSON wastes tokens repeating key names in every row of a tabular array--overhead that scales linearly with row count. This paper presents JTON (JSON Tabular Object Notation), a strict JSON superset whose main idea, Zen Grid, factors column headers into a single row and encodes values with semicolons, preserving JSON's type system while cutting redundancy. Across seven real-world domains, Zen Grid reduces token counts by 15-60% versus JSON compact (28.5% average; 32% with bare_strings). Comprehension tests on 10 LLMs show a net +0.3 pp accuracy gain over JSON: four models improve, three hold steady, and three dip slightly. Generation tests on 12 LLMs yield 100% syntactic validity in both few-shot and zero-shot settings. A Rust/PyO3 reference implementation adds SIMD-accelerated parsing at 1.4x the speed of Python's json module. Code, a 683-vector test suite, and all experimental data are publicly available.