AI Co-Scientist for Ranking: Discovering Novel Search Ranking Models alongside LLM-based AI Agents with Cloud Computing Access

2026-03-23

AI Summary

This work proposes an AI-augmented scientific framework for learning-to-rank research, addressing the field’s heavy reliance on manual effort and lack of automated discovery pipelines. For the first time in ranking systems, we integrate a single LLM agent to execute routine tasks with a multi-LLM consensus mechanism—leveraging GPT-5.2, Gemini Pro 3, and Claude Opus 4.5—for creative ideation and result interpretation. Coupled with cloud-based GPU training orchestration, our system enables end-to-end automation from hypothesis generation and code implementation to model training. The framework autonomously discovers a novel sequential feature processing architecture that significantly outperforms baseline methods on offline metrics, achieving performance comparable to human expert designs while substantially reducing repetitive research labor.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Recent advances in AI agents for software engineering and scientific discovery have demonstrated remarkable capabilities, yet their application to developing novel ranking models in commercial search engines remains unexplored. In this paper, we present an AI Co-Scientist framework that automates the full search ranking research pipeline: from idea generation to code implementation and GPU training job scheduling with expert in the loop. Our approach strategically employs single-LLM agents for routine tasks while leveraging multi-LLM consensus agents (GPT 5.2, Gemini Pro 3, and Claude Opus 4.5) for challenging phases such as results analysis and idea generation. To our knowledge, this is the first study in the ranking community to utilize an AI Co-Scientist framework for algorithmic research. We demonstrate that this framework discovered a novel technique for handling sequence features, with all model enhancements produced automatically, yielding substantial offline performance improvements. Our findings suggest that AI systems can discover ranking architectures comparable to those developed by human experts while significantly reducing routine research workloads.

Problem

• search ranking
• AI agent
• LLM
• algorithmic discovery
• AI co-scientist

Innovation

• AI Co-Scientist
• multi-LLM consensus
• automated ranking research
• sequence feature modeling
• LLM-based scientific discovery