CARMA: Enhanced Compositionality in LLMs via Advanced Regularisation and Mutual Information Alignment

2025-02-16

AI Summary

Large language models (LLMs) exhibit poor compositional generalization, struggling to systematically recombine learned semantics for interpreting novel compositions. To address this, we propose a lightweight, architecture- and data-agnostic intervention method. Our approach introduces three key innovations: (1) mutual information maximization regularization to enhance semantic consistency between inputs and internal representations; (2) joint intra-layer and inter-layer representation stability constraints to mitigate feature fragmentation induced by fine-tuning; and (3) an inverse dictionary modeling framework coupled with perturbation-robustness evaluation. Without compromising downstream task performance, our method significantly improves token-level semantic consistency and compositional reasoning accuracy across multiple benchmarks. It delivers consistent gains across diverse tasks and demonstrates strong cross-architecture transferability—validated on models including LLaMA and Qwen.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Large language models (LLMs) struggle with compositional generalisation, limiting their ability to systematically combine learned components to interpret novel inputs. While architectural modifications, fine-tuning, and data augmentation improve compositionality, they often have limited adaptability, face scalability constraints, or yield diminishing returns on real data. To address this, we propose CARMA, an intervention that enhances the stability and robustness of compositional reasoning in LLMs while preserving fine-tuned performance. CARMA employs mutual information regularisation and layer-wise stability constraints to mitigate feature fragmentation, ensuring structured representations persist across and within layers. We evaluate CARMA on inverse dictionary modelling and sentiment classification, measuring its impact on semantic consistency, performance stability, and robustness to lexical perturbations. Results show that CARMA reduces the variability introduced by fine-tuning, stabilises token representations, and improves compositional reasoning. While its effectiveness varies across architectures, CARMA's key strength lies in reinforcing learned structures rather than introducing new capabilities, making it a scalable auxiliary method. These findings suggest that integrating CARMA with fine-tuning can improve compositional generalisation while maintaining task-specific performance in LLMs.

Problem

• Enhances compositional generalization in LLMs
• Reduces variability from fine-tuning
• Improves robustness and stability in reasoning

Innovation

• Mutual information regularisation mitigates fragmentation
• Layer-wise stability constraints ensure structured representations
• CARMA enhances compositional reasoning in LLMs