AC-LoRA: (Almost) Training-Free Access Control-Aware Multi-Modal LLMs

2025-05-15

AI Summary

To address sensitive data leakage and the absence of fine-grained access control in enterprise multilingual multimodal large language model (LLM) deployment, this paper proposes a permission-aware, training-free LoRA routing framework. Our method dynamically retrieves and weightedly fuses permission-matched LoRA adapters based on similarity between query and document embeddings, enabling cross-modal plug-and-play adaptation and strong information isolation. Critically, it requires no additional routing training; instead, multimodal alignment and permission-aware response generation are inherently embedded within the fusion process. Evaluated on two enterprise-scale datasets, our approach matches or surpasses state-of-the-art LoRA mixture methods in task performance while providing formal guarantees of information isolation. This significantly enhances controllability and practicality of LLMs in high-security operational environments.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Corporate LLMs are gaining traction for efficient knowledge dissemination and management within organizations. However, as current LLMs are vulnerable to leaking sensitive information, it has proven difficult to apply them in settings where strict access control is necessary. To this end, we design AC-LoRA, an end-to-end system for access control-aware corporate LLM chatbots that maintains a strong information isolation guarantee. AC-LoRA maintains separate LoRA adapters for permissioned datasets, along with the document embedding they are finetuned on. AC-LoRA retrieves a precise set of LoRA adapters based on the similarity score with the user query and their permission. This similarity score is later used to merge the responses if more than one LoRA is retrieved, without requiring any additional training for LoRA routing. We provide an end-to-end prototype of AC-LoRA, evaluate it on two datasets, and show that AC-LoRA matches or even exceeds the performance of state-of-the-art LoRA mixing techniques while providing strong isolation guarantees. Furthermore, we show that AC-LoRA design can be directly applied to different modalities.

Problem

• Preventing sensitive data leaks in corporate LLMs with strict access control
• Enabling multi-modal LLMs without additional training for LoRA routing
• Ensuring strong information isolation while maintaining performance

Innovation

• Uses separate LoRA adapters for permissioned datasets
• Retrieves LoRA adapters based on query similarity and permissions
• Merges responses without additional training for LoRA routing