GraphRouter: A Graph-based Router for LLM Selections

2024-10-04

arXiv.org

AI Summary

Existing approaches for dynamic LLM selection in multi-LLM settings suffer from limited generalizability and cost-effectiveness, as they rely on transductive learning and fail to jointly model the intricate interactions among tasks, queries, and LLMs. Method: We propose an inductive heterogeneous graph routing framework that constructs a task-query-LLM heterogeneous graph. It employs edge prediction to jointly model response quality and computational cost, enabling principled multi-objective trade-offs between effectiveness and inference overhead. The framework supports zero-shot adaptation to unseen LLMs without retraining, overcoming the limitations of transductive learning. Contribution/Results: Evaluated across three distinct effectiveness–cost trade-off scenarios, our method achieves average performance gains of ≥12.3%, improves generalization to novel LLMs by ≥9.5%, and significantly reduces inference costs—demonstrating superior scalability, adaptability, and efficiency.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models
• Search and Optimization: Learning to Search

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

The rapidly growing number and variety of Large Language Models (LLMs) present significant challenges in efficiently selecting the appropriate LLM for a given query, especially considering the trade-offs between performance and computational cost. Current LLM selection methods often struggle to generalize across new LLMs and different tasks because of their limited ability to leverage contextual interactions among tasks, queries, and LLMs, as well as their dependence on a transductive learning framework. To address these shortcomings, we introduce a novel inductive graph framework, named as GraphRouter, which fully utilizes the contextual information among tasks, queries, and LLMs to enhance the LLM selection process. GraphRouter constructs a heterogeneous graph comprising task, query, and LLM nodes, with interactions represented as edges, which efficiently captures the contextual information between the query's requirements and the LLM's capabilities. Through an innovative edge prediction mechanism, GraphRouter is able to predict attributes (the effect and cost of LLM response) of potential edges, allowing for optimized recommendations that adapt to both existing and newly introduced LLMs without requiring retraining. Comprehensive experiments across three distinct effect-cost weight scenarios have shown that GraphRouter substantially surpasses existing routers, delivering a minimum performance improvement of 12.3%. In addition, it achieves enhanced generalization across new LLMs settings and supports diverse tasks with at least a 9.5% boost in effect and a significant reduction in computational demands. This work endeavors to apply a graph-based approach for the contextual and adaptive selection of LLMs, offering insights for real-world applications. Our codes for GraphRouter will soon be released at https://github.com/ulab-uiuc/GraphRouter.

Problem

• Efficiently select appropriate LLMs for queries considering performance-cost trade-offs.
• Overcome limitations of current methods in generalizing across new LLMs and tasks.
• Utilize contextual interactions among tasks, queries, and LLMs for enhanced selection.

Innovation

• GraphRouter uses inductive graph framework.
• Heterogeneous graph captures contextual interactions.
• Edge prediction optimizes LLM recommendations.