LEAP: LLM-powered End-to-end Automatic Library for Processing Social Science Queries on Unstructured Data

2025-01-07

AI Summary

Social scientists face significant challenges in efficiently applying machine learning (ML) to analyze unstructured data such as tweets, especially when natural language queries are ambiguous and SQL translation is error-prone. To address this, we propose NL2ML—a novel framework that introduces a fuzzy query filtering mechanism and a dynamic ML function orchestration engine supporting both built-in and user-defined models. We also construct QUIET-ML, the first real-world benchmark of social science research queries (120 instances). NL2ML integrates large language model–driven code generation, ML function registration/calling, ambiguity detection, an end-to-end execution engine, and dynamic table expansion. Evaluated on QUIET-ML, NL2ML achieves 100% Pass@3 and 92% Pass@1, with an average end-to-end cost of just $1.06 (of which code generation accounts for only $0.02), substantially improving analytical accuracy, accessibility, and cost-efficiency for social science researchers.

Technology Category

• Natural Language Processing: Question Answering
• Machine Learning: Privacy
• Data Mining & Knowledge Management: Intelligent Query Processing

Application Category

• User Modeling, Personalization and Recommendation: ML for personalized search and recommendations
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Web Mining and Content Analysis: Machine learning and data science for the Web

Abstract

Social scientists are increasingly interested in analyzing the semantic information (e.g., emotion) of unstructured data (e.g., Tweets), where the semantic information is not natively present. Performing this analysis in a cost-efficient manner requires using machine learning (ML) models to extract the semantic information and subsequently analyze the now structured data. However, this process remains challenging for domain experts. To demonstrate the challenges in social science analytics, we collect a dataset, QUIET-ML, of 120 real-world social science queries in natural language and their ground truth answers. Existing systems struggle with these queries since (1) they require selecting and applying ML models, and (2) more than a quarter of these queries are vague, making standard tools like natural language to SQL systems unsuited. To address these issues, we develop LEAP, an end-to-end library that answers social science queries in natural language with ML. LEAP filters vague queries to ensure that the answers are deterministic and selects from internally supported and user-defined ML functions to extend the unstructured data to structured tables with necessary annotations. LEAP further generates and executes code to respond to these natural language queries. LEAP achieves a 100% pass @ 3 and 92% pass @ 1 on QUIET-ML, with a $1.06 average end-to-end cost, of which code generation costs $0.02.

Problem

• Machine Learning
• Natural Language Processing
• Data Analysis Efficiency

Innovation

• Superlanguage Capability
• Automated Machine Learning Model Application
• Unstructured Data to Structured Insights