Autonomy-Aware Clustering: When Local Decisions Supersede Global Prescriptions

2025-09-30

AI Summary

In real-world clustering, entities often exhibit local autonomy, deviating from global assignments and causing dynamic changes in cluster structure, composition, and size—significantly degrading downstream inference. To address this, we propose the first *autonomy-aware clustering* framework. Our method models entity-level decision-making as a reinforcement learning process, integrated with deterministic annealing—leveraging phase-transition properties to design an efficient annealing schedule—and ADEN, a transferable, variable-length-input-compatible attention network built upon the Transformer architecture. ADEN enables dynamic dependency modeling and cross-instance knowledge transfer. Experiments demonstrate that, without explicit autonomy modeling, our approach achieves only a 3–4% deviation from ground-truth distributions—outperforming conventional autonomy-agnostic methods by 35–40%. This substantially improves both fidelity to real-world behavioral patterns and robustness under distributional shifts.

Technology Category

• Machine Learning: Clustering
• Multiagent Systems: Adversarial Agents
• Search and Optimization: Learning to Search

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization

Abstract

Clustering arises in a wide range of problem formulations, yet most existing approaches assume that the entities under clustering are passive and strictly conform to their assigned groups. In reality, entities often exhibit local autonomy, overriding prescribed associations in ways not fully captured by feature representations. Such autonomy can substantially reshape clustering outcomes -- altering cluster compositions, geometry, and cardinality -- with significant downstream effects on inference and decision-making. We introduce autonomy-aware clustering, a reinforcement (RL) learning framework that learns and accounts for the influence of local autonomy without requiring prior knowledge of its form. Our approach integrates RL with a deterministic annealing (DA) procedure, where, to determine underlying clusters, DA naturally promotes exploration in early stages of annealing and transitions to exploitation later. We also show that the annealing procedure exhibits phase transitions that enable design of efficient annealing schedules. To further enhance adaptability, we propose the Adaptive Distance Estimation Network (ADEN), a transformer-based attention model that learns dependencies between entities and cluster representatives within the RL loop, accommodates variable-sized inputs and outputs, and enables knowledge transfer across diverse problem instances. Empirical results show that our framework closely aligns with underlying data dynamics: even without explicit autonomy models, it achieves solutions close to the ground truth (gap ~3-4%), whereas ignoring autonomy leads to substantially larger gaps (~35-40%). The code and data are publicly available at https://github.com/salar96/AutonomyAwareClustering.

Problem

• Modeling entity autonomy that overrides prescribed clustering assignments
• Developing RL framework to capture autonomy without prior knowledge
• Addressing autonomy's impact on cluster composition and cardinality

Innovation

• Reinforcement learning framework for local autonomy
• Deterministic annealing with phase transition scheduling
• Transformer-based adaptive distance estimation network