ChamaleonLLM: Batch-Aware Dynamic Low-Rank Adaptation via Inference-Time Clusters

2025-02-06

AI Summary

Existing large language models (LLMs) rely on static, fixed-weight parameters during inference, limiting their adaptability to dynamic data distributions. To address this, we propose Batch-Aware Dynamic Low-Rank Adaptation (BDLA): a lightweight hypernetwork dynamically clusters input batches based on semantic similarity and generates context-aware, low-rank weight updates tailored to each cluster; these updates are seamlessly fused into the backbone model at inference time. Unlike prior approaches, BDLA eliminates predefined experts or static LoRA configurations, enabling fully parameter-free, plug-and-play inference-time adaptation. Evaluated across diverse multi-task benchmarks, BDLA achieves consistent improvements of 2.1–4.7 percentage points in average accuracy, incurs zero additional expert-model overhead, and introduces less than 8% latency overhead—demonstrating efficient, adaptive inference without architectural or parametric expansion.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Recent advances in large language models (LLMs) have shown remarkable performance across diverse tasks. However, these models are typically deployed with fixed weights, which limits their ability to adapt dynamically to the variability inherent in real-world data during inference. This paper introduces ChamaleonLLM, a novel framework that enables inference-time adaptation of LLMs by leveraging batch-aware clustering and on-the-fly generation of low-rank updates. Unlike traditional fine-tuning approaches such as Low-Rank Adaptation (LoRA) or methods that rely on a fixed set of pre-learned uniforms (changeable masks), our method dynamically generates adaptive modifications to the decoder weights based on the aggregated statistics of clustered batches. By intelligently grouping similar inputs and computing context-aware low-rank updates via a hyper-network, ChamaleonLLM achieves significant performance gains, outperforming conventional LoRA methods while eliminating the overhead of maintaining multiple expert models. Our experiments highlight the potential of our approach to serve as a versatile and highly adaptive solution for language model inference. ChamaleonLLM is open-sourced to ensure the reproducibility of our experiments: https://anonymous.4open.science/r/ChamaleonLLM/

Problem

• Dynamic adaptation of LLMs during inference
• Batch-aware clustering for input grouping
• On-the-fly low-rank weight updates

Innovation

• Dynamic low-rank adaptation during inference
• Batch-aware clustering for input grouping
• Hyper-network generates context-aware updates