Problem
Research questions and friction points this paper is trying to address.
Just-in-Time Systems
System Synthesis
Key-Value Stores
LLM-based Coding Agents
Performance Specialization
The Time is Here for Just-in-Time Systems: Challenges and Opportunities
🤖 AI Summary
Traditional core systems, such as key-value stores, are difficult to optimize for specific workloads and deployment environments due to their generic design and lengthy development cycles. This work proposes Jitskit, the first end-to-end just-in-time system synthesis framework powered by large language models (LLMs), which employs LLM-based coding agents to generate fully customized systems from scratch. Functional correctness is ensured through specification cards, iterative refinement, and dynamic test suites. By circumventing the conventional paradigm that relies on years of manual engineering, Jitskit outperforms state-of-the-art systems across 18 configurations, achieving up to a 4.6× performance improvement. Moreover, compared to direct use of Claude Code, it effectively avoids reward-hacking behaviors and yields performance gains of up to 5.4×.
📝 Abstract
Core systems like key-value stores have historically taken years to build, and are designed to be general so as to amortize cost across deployments, paying a significant performance cost. We argue that LLM-based coding agents now make a different approach tractable: Just-in-Time Systems, in which the entire system is synthesized from scratch, specialized to the environment, workload, and required system properties. We present a JIT system synthesis pipeline, Jitskit, and explore its effectiveness in synthesizing key-value stores from spec cards that span different YCSB workloads, deployment constraints (e.g., compute resources), and system properties (e.g., consistency and durability). Jitskit iteratively refines a system implementation to match the specification against an evolving evaluation test suite. The resulting synthesized systems are performant, beating comparable state-of-the-art systems on 18 of 18 specs tried, by up to 4.6x over the best off-the-shelf baseline on the most favorable spec. Naively running Claude Code either reward-hacks or underperforms Jitskit by up to 5.4x. We discuss the challenges we overcame in building Jitskit and our key takeaways.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Just-in-Time Systems
LLM-based coding agents
system synthesis
key-value stores
specification-driven optimization
🔎 Similar Papers
No similar papers found.
💼 Related Jobs
