HiCoLoRA: Addressing Context-Prompt Misalignment via Hierarchical Collaborative LoRA for Zero-Shot DST

2025-09-23

AI Summary

To address semantic misalignment between dynamic dialogue contexts and static prompts in zero-shot dialogue state tracking (zs-DST)—which causes rigid cross-layer coordination, domain interference, and catastrophic forgetting—this paper proposes the Hierarchical Collaborative Low-Rank Adaptation (HCLA) framework. HCLA constructs a hierarchical LoRA architecture that jointly integrates spectral clustering–driven domain-slot association discovery, semantics-aware SVD-based initialization, and an adaptive linear fusion mechanism. This design enables parameter-efficient dynamic prompt alignment and cross-layer semantic collaboration. Evaluated on MultiWOZ and SGD benchmarks under strict zero-shot settings, HCLA achieves state-of-the-art performance for the first time, significantly improving cross-domain generalization and robustness without any task-specific fine-tuning or in-domain data.

Technology Category

• Natural Language Processing: Conversational AI/Dialog Systems
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Zero-shot Dialog State Tracking (zs-DST) is essential for enabling Task-Oriented Dialog Systems (TODs) to generalize to new domains without costly data annotation. A central challenge lies in the semantic misalignment between dynamic dialog contexts and static prompts, leading to inflexible cross-layer coordination, domain interference, and catastrophic forgetting. To tackle this, we propose Hierarchical Collaborative Low-Rank Adaptation (HiCoLoRA), a framework that enhances zero-shot slot inference through robust prompt alignment. It features a hierarchical LoRA architecture for dynamic layer-specific processing (combining lower-layer heuristic grouping and higher-layer full interaction), integrates Spectral Joint Domain-Slot Clustering to identify transferable associations (feeding an Adaptive Linear Fusion Mechanism), and employs Semantic-Enhanced SVD Initialization (SemSVD-Init) to preserve pre-trained knowledge. Experiments on multi-domain datasets MultiWOZ and SGD show that HiCoLoRA outperforms baselines, achieving SOTA in zs-DST. Code is available at https://github.com/carsonz/HiCoLoRA.

Problem

• Addressing semantic misalignment between dialog contexts and static prompts
• Enhancing zero-shot slot inference through robust prompt alignment
• Preventing domain interference and catastrophic forgetting in DST

Innovation

• Hierarchical LoRA architecture for layer-specific processing
• Spectral clustering to identify transferable domain-slot associations
• Semantic-enhanced SVD initialization to preserve pretrained knowledge