ScriptoriumWS: A Code Generation Assistant for Weak Supervision

2025-02-17

AI Summary

To address the high cost and low coverage of manually crafting high-quality labeling functions (LFs) in weak supervision, this paper pioneers the use of large language models (LLMs) as coding assistants, proposing a multi-level prompting strategy to automatically generate domain-adapted, noisy LFs. Our method jointly integrates handcrafted and LLM-generated LFs, leveraging weak supervision label aggregation (e.g., Snorkel) and a rule quality assessment mechanism to achieve optimal fusion. Experiments across multiple real-world tasks show that our approach matches human-written LFs in pseudo-label accuracy while substantially improving LF coverage (+38.2% on average) and reducing expert effort (62% fewer manual rules). The core contribution is the establishment of a novel LLM-augmented paradigm for weak supervision rule engineering—featuring an interpretable, integrable generation-and-enhancement framework that bridges human expertise and generative AI.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Weak supervision is a popular framework for overcoming the labeled data bottleneck: the need to obtain labels for training data. In weak supervision, multiple noisy-but-cheap sources are used to provide guesses of the label and are aggregated to produce high-quality pseudolabels. These sources are often expressed as small programs written by domain experts -- and so are expensive to obtain. Instead, we argue for using code-generation models to act as coding assistants for crafting weak supervision sources. We study prompting strategies to maximize the quality of the generated sources, settling on a multi-tier strategy that incorporates multiple types of information. We explore how to best combine hand-written and generated sources. Using these insights, we introduce ScriptoriumWS, a weak supervision system that, when compared to hand-crafted sources, maintains accuracy and greatly improves coverage.

Problem

• Overcomes data labeling bottleneck using weak supervision.
• Employs code-generation models for creating supervision sources.
• Combines hand-written and generated sources effectively.

Innovation

• Code-generation models assist weak supervision
• Multi-tier prompting enhances source quality
• Combines hand-written and generated sources effectively