Large Databases Need Small, Open-Weight Language Models

2026-06-30

AI Summary

This work addresses the prohibitive cost of closed-source large language model (LLM) APIs in database-augmented applications—exceeding $10,000 per experiment—which severely hinders both research and deployment. To overcome this limitation, the authors propose a lightweight alternative that integrates quantized open-source small language models with inference optimizations and the newly introduced BlendSQL v0.1.0 framework, enabling efficient database interaction on local hardware with only 16GB of GPU memory. This approach is the first to systematically demonstrate that compact open-source models can achieve not only comparable or superior accuracy but also significantly lower latency (3.8× faster) and substantially higher cost efficiency—reducing overall expenses by a factor of 390 compared to closed-source APIs—thereby challenging the prevailing assumption that proprietary models are indispensable for effective LM-DB integration.

Abstract

Language model systems built around proprietary APIs often operate on a token-based cost model. This becomes prohibitively expensive in the context of large databases, where LM-enhanced relational operators can incur costs exceeding $10,000 for a single set of experiments, hindering thorough research and practical deployment. In this paper, we demonstrate that quantized, open-weight models running locally on just 16GB of VRAM can match or exceed the accuracy of closed-source counterparts at lower latency and a fraction of the price, challenging the prevailing assumption that closed-source LM APIs are necessary for effective LM-database integration. We present and analyze the key system optimizations required to efficiently deploy these open-weight models within an LM-DB system. By integrating these local models into the BlendSQL v0.1.0 framework, we demonstrate a 390x reduction in overall costs and 3.8x reduction in latency compared to a proprietary LM API. We make our code available at https://github.com/CapitalOne-Research/play-by-the-type-rules/tree/main/sembench.

Problem

• large databases
• language models
• cost efficiency
• proprietary APIs
• LM-database integration

Innovation

• open-weight language models
• quantization
• LM-DB integration
• cost-efficient inference
• local deployment