Training Large Language Models to be Better Rule Followers

2025-02-17

AI Summary

Large language models (LLMs) often fail on simple rule-based reasoning tasks despite strong performance on other benchmarks, revealing an overreliance on case-based analogy rather than strict rule execution—stemming from the absence of explicit modeling of rule-instance mappings during pretraining. Method: We introduce the first comprehensive rule-following evaluation benchmark covering 88 cross-domain tasks and propose Meta-RFFT (Meta Rule-Following Fine-Tuning), a novel framework that systematically models structured rule-instance relationships via rule-driven data construction and meta-fine-tuning. Contribution/Results: Experiments demonstrate that Meta-RFFT significantly improves generalization under few-shot prompting and downstream fine-tuning, consistently outperforming all baselines across diverse rule-following tasks. Our results confirm that large-scale, structured rule training substantially enhances LLMs’ ability to execute abstract rules strictly and consistently.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Metareasoning and Metaheuristics
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Large language models (LLMs) have shown impressive performance across a wide range of tasks. However, they often exhibit unexpected failures in seemingly straightforward tasks, suggesting a reliance on case-based reasoning rather than rule-based reasoning. While the vast training corpus of LLMs contains numerous textual"rules", current training methods fail to leverage these rules effectively. Crucially, the relationships between these"rules"and their corresponding"instances"are not explicitly modeled. As a result, while LLMs can often recall rules with ease, they fail to apply these rules strictly and consistently in relevant reasoning scenarios. In this paper, we investigate the rule-following capabilities of LLMs and propose Meta Rule-Following Fine-Tuning (Meta-RFFT) to enhance the cross-task transferability of rule-following abilities. We first construct a dataset of 88 tasks requiring following rules, encompassing diverse reasoning domains. We demonstrate through extensive experiments that models trained on large-scale rule-following tasks are better rule followers, outperforming the baselines in both downstream fine-tuning and few-shot prompting scenarios. This highlights the cross-task transferability of models with the aid of Meta-RFFT. Furthermore, we examine the influence of factors such as dataset size, rule formulation, and in-context learning.

Problem

• Enhance rule-following in language models
• Improve cross-task rule application transferability
• Address failures in rule-based reasoning tasks

Innovation

• Meta Rule-Following Fine-Tuning
• Cross-task transferability enhancement
• Large-scale rule-following tasks dataset