Rethinking Foundation Model Collaboration: Enhancing Specialized Models through Proxy Task Reasoning

2026-06-30

AI Summary

This work addresses the limitations of foundation models in structured prediction tasks, where their geometric and numerical estimation capabilities often fall short of those of specialized models. To bridge this gap, the authors propose the FAT framework, which decomposes the task into three stages: candidate hypothesis generation by a task-specific model, reconstruction of a multimodal information space, and bounded proxy reasoning—such as selection or verification—performed by a foundation model. This approach innovatively casts collaboration as a proxy task, preserving the inherent task structure while leveraging the contextual understanding of foundation models. Experiments demonstrate that FAT consistently outperforms specialized baselines across diverse tasks—including 2D/3D object detection, trajectory prediction, and semantic segmentation—and significantly surpasses direct regression with foundation models at substantially lower computational cost.

Abstract

Foundation models are increasingly integrated into embodied intelligence systems, but directly assigning them structured prediction tasks requires precise geometric and numerical estimation, where specialized models often remain stronger. This capability mismatch raises a key question: should foundation models replace task-specific predictors, or should they collaborate through tasks better aligned with their strengths? We propose FAT, a foundation-model-augmented task-specific reasoning framework that treats collaboration as task decomposition rather than model replacement. FAT decomposes structured prediction into specialist prediction, information-space reconstruction, and foundation-model proxy reasoning. The specialist generates geometrically and physically valid hypotheses in the native output space, while the foundation model performs a bounded proxy task, such as selection or verification, over reconstructed multimodal candidates. We instantiate this principle as ProxySelect with a vision--language model. Across 2D object detection, 3D object detection, trajectory prediction, and semantic segmentation, ProxySelect consistently improves specialized baselines and substantially outperforms direct foundation-model regression at lower computational cost. These results suggest a general collaboration principle: specialized models preserve task-specific structure, while foundation models refine their hypotheses through contextual proxy reasoning.

Problem

• foundation models
• specialized models
• structured prediction
• model collaboration
• proxy reasoning

Innovation

• foundation model collaboration
• task decomposition
• proxy task reasoning
• specialized models
• structured prediction