RuleChef: Grounding LLM Task Knowledge in Human-Editable Rules

2026-07-01

AI Summary

This work proposes a novel method for distilling the implicit knowledge of large language models (LLMs) into editable and interpretable rule-based systems. By leveraging task descriptions and annotated examples, the approach guides an LLM to generate initial executable rules, which are then iteratively refined through additional samples and human feedback; alternatively, rules can be automatically constructed from input–output pairs of an existing model. This method represents the first effective framework for efficiently translating LLM behavior into human-readable rules, enabling a human-in-the-loop, closed-loop optimization process. Empirical validation on text classification and named entity recognition tasks demonstrates its efficacy. The open-sourced tool RuleChef produces rule systems that are not only efficient and deterministic but also fully auditable.

Abstract

We present RuleChef, a framework that uses large language models (LLMs) to generate executable rules for NLP tasks such as text classification, Named Entity Recognition (NER), or relation extraction. Rules are generated based on a task description and a set of labeled examples, then they are iteratively improved based both on additional examples and on human feedback overexisting rules. RuleChef can also be used to bootstrap rules using the observed input-output pairs from any existing model for a given task. LLMs are used only at learning time, synthesizing rules and iteratively patching them based on failures measured on a held-out split. The result of this process is a fast, deterministic, and inspectable rule system. Preliminary evaluation is performed on both classification and NER tasks. We release RuleChef as open-source software under an Apache 2.0

Problem

• large language models
• rule-based systems
• natural language processing
• human-editable rules
• knowledge grounding

Innovation

• Rule-based NLP
• LLM-guided rule synthesis
• Human-in-the-loop learning
• Interpretable AI
• Rule bootstrapping