Mixing Configurations for Downstream Prediction

2025-10-22

AI Summary

This work addresses the challenge of adaptively leveraging unsupervised hierarchical clustering configurations for downstream predictive tasks. We propose GraMixC, a plug-and-play module that (i) formally defines the “configuration” concept, (ii) introduces a Reverse Merge/Split alignment mechanism to enable task-driven dynamic fusion of multi-scale features, and (iii) generates cross-resolution clustering configurations via community detection and self-supervised learning—guided by the similarity structure of Vision Transformer register tokens—and integrates features efficiently through attention heads. GraMixC operates fully unsupervised and requires no manual hyperparameter tuning. On the DSN1 16S rRNA prediction task, it achieves an R² of 0.9 (+0.3 over prior SOTA), establishing new state-of-the-art performance. Moreover, it consistently outperforms both single-scale and static-feature baselines across multiple tabular-data benchmarks.

Technology Category

• Machine Learning: Unsupervised & Self-Supervised Learning
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Vertical and domain-specific search

Abstract

Humans possess an innate ability to group objects by similarity, a cognitive mechanism that clustering algorithms aim to emulate. Recent advances in community detection have enabled the discovery of configurations -- valid hierarchical clusterings across multiple resolution scales -- without requiring labeled data. In this paper, we formally characterize these configurations and identify similar emergent structures in register tokens within Vision Transformers. Unlike register tokens, configurations exhibit lower redundancy and eliminate the need for ad hoc selection. They can be learned through unsupervised or self-supervised methods, yet their selection or composition remains specific to the downstream task and input. Building on these insights, we introduce GraMixC, a plug-and-play module that extracts configurations, aligns them using our Reverse Merge/Split (RMS) technique, and fuses them via attention heads before forwarding them to any downstream predictor. On the DSN1 16S rRNA cultivation-media prediction task, GraMixC improves the R2 score from 0.6 to 0.9 across multiple methods, setting a new state of the art. We further validate GraMixC on standard tabular benchmarks, where it consistently outperforms single-resolution and static-feature baselines.

Problem

• Discovering hierarchical clustering configurations without labeled data
• Reducing redundancy in register tokens for Vision Transformers
• Improving downstream task performance through configuration fusion

Innovation

• GraMixC extracts hierarchical configurations from data
• Aligns configurations using Reverse Merge/Split technique
• Fuses configurations via attention heads for prediction