Problem
Research questions and friction points this paper is trying to address.
autonomous research
open-ended problems
telecom ticket retrieval
machine learning
Autonomous Research for Open-Ended Problems: A Case Study on Telecom Ticket Retrieval
🤖 AI Summary
本文探讨了使用自主研究解决开放性问题，如电信票证检索，并发现结合商业和开源代理的自主研究在狭窄超参数优化上表现出色，但在直觉和创造力方面不足。
📝 Abstract
Recent breakthroughs in LLM-based systems and their abilities in problem solving and coding have allowed progress in the AI for Science paradigm, potentially replacing human roles in machine learning (ML) research. However, while several frameworks of fully autonomous end-to-end ML research have been proposed, successful implementations of them are often limited to problems with narrow search spaces, like language modeling or biomedical ML benchmarks. In this paper, we explore how autonomous research can be adapted to solve open-ended, industry-grade ML problems, by considering a case study: telecom ticket retrieval, an open-ended task with degrees of freedom in representation, architecture, and training data generation. We discover that autonomous research for open-ended problems with commercial and open-source agents shows both promise and limitations: while autonomous research can excel in narrow hyperparameter optimization, it lacks human-like intuition and creativity and requires operational overhead. Even with minimal human supervision, autonomous research can reach $90\%$ of state-of-the-art performance (0.34 vs. 0.38 Recall@1) in a much shorter time period (10 weeks vs. 10 months of human work) at a modest cost (up to \$200 per Cursor campaign). Our empirical evidence recommends that human researchers and autonomous research frameworks work together for best results in ML research.
Innovation
Methods, ideas, or system contributions that make the work stand out.
autonomous research
open-ended problems
telecom ticket retrieval
human supervision
🔎 Similar Papers
No similar papers found.
