Large Language Models Orchestrating Structured Reasoning Achieve Kaggle Grandmaster Level

2024-11-05

arXiv.org

AI Summary

Current large language model (LLM)-based agents lack human-like continual learning capabilities, hindering dynamic adaptation through interaction, reflection, and internal model updates. To address this, we propose Agent K v1.0—the first end-to-end autonomous data science agent capable of full-pipeline Kaggle competition automation, from URL input to submission, supporting tabular, computer vision (CV), natural language processing (NLP), and multimodal tasks. Methodologically, we introduce a nested structured reasoning framework enabling experience-driven optimization without backpropagation, and a reward-guided memory selection mechanism that dynamically regulates long- and short-term memory. We further integrate Bayesian optimization, experience-replay-based memory management, and the Elo-MMR evaluation framework. Empirically, Agent K v1.0 achieves a 92.5% task success rate across diverse Kaggle benchmarks and ranks in the top 38% under Elo-MMR—matching the performance of human Grandmasters (6 gold, 3 silver, 7 bronze medals).

Technology Category

• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Multiagent Systems: Multiagent Learning
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Agentic search

Abstract

We introduce Agent K v1.0, an end-to-end autonomous data science agent designed to automate, optimise, and generalise across diverse data science tasks. Fully automated, Agent K v1.0 manages the entire data science life cycle by learning from experience. It leverages a highly flexible structured reasoning framework to enable it to dynamically process memory in a nested structure, effectively learning from accumulated experience stored to handle complex reasoning tasks. It optimises long- and short-term memory by selectively storing and retrieving key information, guiding future decisions based on environmental rewards. This iterative approach allows it to refine decisions without fine-tuning or backpropagation, achieving continuous improvement through experiential learning. We evaluate our agent's apabilities using Kaggle competitions as a case study. Following a fully automated protocol, Agent K v1.0 systematically addresses complex and multimodal data science tasks, employing Bayesian optimisation for hyperparameter tuning and feature engineering. Our new evaluation framework rigorously assesses Agent K v1.0's end-to-end capabilities to generate and send submissions starting from a Kaggle competition URL. Results demonstrate that Agent K v1.0 achieves a 92.5% success rate across tasks, spanning tabular, computer vision, NLP, and multimodal domains. When benchmarking against 5,856 human Kaggle competitors by calculating Elo-MMR scores for each, Agent K v1.0 ranks in the top 38%, demonstrating an overall skill level comparable to Expert-level users. Notably, its Elo-MMR score falls between the first and third quartiles of scores achieved by human Grandmasters. Furthermore, our results indicate that Agent K v1.0 has reached a performance level equivalent to Kaggle Grandmaster, with a record of 6 gold, 3 silver, and 7 bronze medals, as defined by Kaggle's progression system.

Problem

• Designing LLM agents with structured human-like experiential learning
• Enabling autonomous agents to master complex tasks through cognitive frameworks
• Achieving human-level performance in data science competitions via automated learning

Innovation

• Kolb-based experiential learning cycle for agents
• Separation of extrinsic and intrinsic cognitive functions
• Autonomous data science code generation achieving human-level performance