Compositional Context Fine-Tuning Vision-Language Model for Complex Assembly Action Understanding from Videos

2026-07-12

AI Summary

Understanding complex assembly actions remains highly challenging due to fine-grained hand-object interactions and subtle motion distinctions, hindering efficient human-robot collaboration. This work proposes a Compositional Contextual Fine-Tuning (CCFT) framework that decomposes actions into semantic primitives—such as verbs, objects, and tools—and leverages templated visual question answering (VQA) pairs to fine-tune vision-language models for element-level recognition. To mitigate multi-task interference under data scarcity and enable independent hyperparameter tuning, the authors further introduce a Layer-wise Progressive Alternating Training (LP-AT) strategy, which alternately optimizes task-specific low-rank adapters. Evaluated on the newly curated HA-ViD-VQA and IKEA-ASM-VQA datasets, the method substantially outperforms strong baselines, yielding near-deterministic and interpretable predictions suitable for diverse downstream applications.

Abstract

Assembly action understanding is a key enabler for effective human-robot collaborative assembly, yet it remains challenging due to subtle motions and fine-grained hand-object interactions. We adapt vision-language models (VLMs) to this challenging domain with Compositional Context Fine-Tuning (CCFT), a method that decomposes assembly actions into semantic elements (Verb, Object, Tool) and fine-tunes VLMs to recognize each action element using templated question-answering pairs. This approach ensures near-deterministic outputs. To enable efficient and effective multi-task learning under limited data, a Layer-Partitioned Alternating Training (LP-AT) method is presented, which assigns distinct model layers to recognize specific action elements through element-specific low-rank adapters. LP-AT alternates weight updates across element-specific adapters, reducing cross-task interference while enabling per-adapter hyperparameter optimization. Furthermore, we create HA-ViD-VQA and IKEA-ASM-VQA datasets from existing assembly video datasets. Extensive experiments on these datasets demonstrate that our method consistently outperforms strong action recognition baselines while providing interpretable element-level predictions that can support diverse downstream applications.

Problem

• assembly action understanding
• vision-language models
• fine-grained hand-object interactions
• complex assembly actions
• video understanding

Innovation

• Compositional Context Fine-Tuning
• Vision-Language Model
• Layer-Partitioned Alternating Training
• Assembly Action Understanding
• Low-Rank Adapters