MachineLearningLM: Continued Pretraining Language Models on Millions of Synthetic Tabular Prediction Tasks Scales In-Context ML

2025-09-08

AI Summary

Large language models (LLMs) exhibit diminishing returns in in-context learning (ICL) as the number of demonstration examples increases, particularly on structured tabular prediction tasks. To address this, we propose a portable continual pretraining framework that optimizes Qwen-2.5-7B-Instruct using millions of synthetic tabular tasks generated from structural causal models. Our method integrates random forest–guided teacher distillation, efficient token serialization for tabular data, and LoRA-based adaptation. It is the first to achieve multi-example scaling laws in ICL—where accuracy monotonically improves with increasing context examples—without task-specific fine-tuning, matching random forest performance out-of-the-box. Evaluated across financial, physical, biological, and medical tabular classification benchmarks, our model outperforms strong baselines by 15% on average, increases effective context capacity by 3–6×, boosts batched inference throughput by 50×, and achieves 75.4% on MMLU.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Large language models (LLMs) possess broad world knowledge and strong general-purpose reasoning ability, yet they struggle to learn from many in-context examples on standard machine learning (ML) tasks, that is, to leverage many-shot demonstrations purely via in-context learning (ICL) without gradient descent. We introduce MachineLearningLM, a portable continued-pretraining framework that equips a general-purpose LLM with robust in-context ML capability while preserving its general knowledge and reasoning for broader chat workflows. Our pretraining procedure synthesizes ML tasks from millions of structural causal models (SCMs), spanning shot counts up to 1,024. We begin with a random-forest teacher, distilling tree-based decision strategies into the LLM to strengthen robustness in numerical modeling. All tasks are serialized with a token-efficient prompt, enabling 3x to 6x more examples per context window and delivering up to 50x amortized throughput via batch inference. Despite a modest setup (Qwen-2.5-7B-Instruct with LoRA rank 8), MachineLearningLM outperforms strong LLM baselines (e.g., GPT-5-mini) by an average of about 15% on out-of-distribution tabular classification across finance, physics, biology, and healthcare domains. It exhibits a striking many-shot scaling law: accuracy increases monotonically as in-context demonstrations grow from 8 to 1,024. Without any task-specific training, it attains random-forest-level accuracy across hundreds of shots. General chat capabilities, including knowledge and reasoning, are preserved: it achieves 75.4% on MMLU.

Problem

• Enhancing in-context learning for tabular prediction tasks
• Scaling many-shot demonstrations without gradient descent
• Preserving general knowledge while improving ML capability

Innovation

• Continued pretraining with synthetic tabular tasks
• Distilling tree-based strategies into language model
• Token-efficient serialization enabling batch inference throughput