AIvailable: A Software-Defined Architecture for LLM-as-a-Service on Heterogeneous and Legacy GPUs
Deploying high-performance large language model (LLM) inference services remains challenging in resource-constrained environments—such as academic labs and SMEs—especially when leveraging heterogeneous, legacy GPU hardware (NVIDIA/AMD). Method: This paper proposes a low-overhead LLM-as-a-Service platform designed for heterogeneous legacy GPUs. It employs a software-defined AI architecture featuring a novel VRAM-aware dynamic model allocation and rescheduling mechanism, abstracting hardware heterogeneity to enable fully GPU-accelerated, CPU-fallback-free cross-vendor inference. The platform integrates secure request routing, lightweight load balancing, and distributed monitoring. Contribution/Results: To our knowledge, this is the first system enabling highly available, elastic inference of open-source LLMs on mixed legacy GPU clusters, significantly improving VRAM utilization. It offers a unified API supporting multiple models, lowering deployment barriers. Experiments show a 2.3× increase in VRAM utilization over baseline approaches, 99.8% service availability, and substantial reduction in LLMaaS operational cost.