Arming Data Agents with Tribal Knowledge

2026-02-13

AI Summary

This work addresses the challenge that NL2SQL agents often produce erroneous queries when handling large-scale real-world databases due to insufficient deep semantic understanding of the data. To mitigate this, the authors propose the Tk-Boost framework, which introduces the novel concept of “tribal knowledge.” By analyzing errors from a small set of initial queries, Tk-Boost automatically generates corrective knowledge pieces annotated with applicability conditions. During subsequent query processing, it dynamically retrieves and applies this knowledge in a conditional manner to provide targeted feedback, thereby rectifying semantic misinterpretations on the fly. Evaluated on the Spider 2.0 and BIRD benchmarks, the approach significantly enhances translation accuracy, improving execution accuracy by up to 16.9% and 13.7%, respectively, across multiple NL2SQL agents.

Technology Category

• Data Mining & Knowledge Management: Intelligent Query Processing
• Natural Language Processing: Question Answering
• Knowledge Representation and Reasoning: Knowledge Acquisition

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Agentic search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Natural language to SQL (NL2SQL) translation enables non-expert users to query relational databases through natural language. Recently, NL2SQL agents, powered by the reasoning capabilities of Large Language Models (LLMs), have significantly advanced NL2SQL translation. Nonetheless, NL2SQL agents still make mistakes when faced with large-scale real-world databases because they lack knowledge of how to correctly leverage the underlying data (e.g., knowledge about the intent of each column) and form misconceptions about the data when querying it, leading to errors. Prior work has studied generating facts about the database to provide more context to NL2SQL agents, but such approaches simply restate database contents without addressing the agent's misconceptions. In this paper, we propose Tk-Boost, a bolt-on framework for augmenting any NL2SQL agent with tribal knowledge: knowledge that corrects the agent's misconceptions in querying the database accumulated through experience using the database. To accumulate experience, Tk-Boost first asks the NL2SQL agent to answer a few queries on the database, identifies the agent's misconceptions by analyzing its mistakes on the database, and generates tribal knowledge to address them. To enable accurate retrieval, Tk-Boost indexes this knowledge with applicability conditions that specify the query features for which the knowledge is useful. When answering new queries, Tk-Boost uses this knowledge to provide feedback to the NL2SQL agent, resolving the agent's misconceptions during SQL generation, and thus improving the agent's accuracy. Extensive experiments across the BIRD and Spider 2.0 benchmarks with various NL2SQL agents shows Tk-Boost improves NL2SQL agents accuracy by up to 16.9% on Spider 2.0 and 13.7% on BIRD

Problem

• NL2SQL
• tribal knowledge
• misconceptions
• relational databases
• Large Language Models

Innovation

• tribal knowledge
• NL2SQL
• misconception correction
• LLM-based agents
• contextual feedback