Doing More with Less -- Implementing Routing Strategies in Large Language Model-Based Systems: An Extended Survey
Large language model (LLM) systems commonly suffer from substantial resource waste due to static or suboptimal deployment strategies. Method: This paper proposes a cost-quality–aware query routing mechanism that dynamically dispatches user queries to the most suitable lightweight model, domain-specific expert, or embedding strategy. We formally define the routing problem for the first time and introduce a novel taxonomy jointly optimizing relevance and resource efficiency. Our framework systematically compares academic approaches with industrial practices, incorporating query understanding, policy selection, multi-granularity routing (at both model and embedding levels), fine-grained cost modeling, and a unified evaluation protocol. Contribution/Results: Experiments demonstrate that our mechanism significantly reduces inference overhead—by up to 42% in latency and 38% in GPU memory—while maintaining or even improving answer quality across diverse benchmarks. This work establishes both a theoretical foundation and a reproducible, practical paradigm for building efficient, scalable, and cost-effective LLM systems.