FlexiAct: Towards Flexible Action Control in Heterogeneous Scenarios

2025-05-06

AI Summary

This work addresses action transfer across heterogeneous scenarios, where layout, viewpoint, and skeletal structure discrepancies exist between the target image and reference video, while preserving target identity fidelity. To this end, we propose RefAdapter—a lightweight image-conditioned adapter—and a Frequency-Aware Action Extraction (FAE) mechanism that explicitly disentangles low-frequency action semantics from high-frequency appearance details during the diffusion model’s denoising process, thereby relaxing conventional rigid constraints on pose, viewpoint, and structural alignment. Furthermore, we introduce a spatial-structure adaptive module and an identity-consistency loss to enhance cross-domain generalization. Experiments demonstrate that our method achieves high-fidelity, highly controllable action transfer across diverse skeletons, viewpoints, and compositions, significantly improving both action accuracy and appearance consistency. The code and models are publicly available.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Cognitive Modeling & Cognitive Systems: Adaptive Behavior

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Action customization involves generating videos where the subject performs actions dictated by input control signals. Current methods use pose-guided or global motion customization but are limited by strict constraints on spatial structure, such as layout, skeleton, and viewpoint consistency, reducing adaptability across diverse subjects and scenarios. To overcome these limitations, we propose FlexiAct, which transfers actions from a reference video to an arbitrary target image. Unlike existing methods, FlexiAct allows for variations in layout, viewpoint, and skeletal structure between the subject of the reference video and the target image, while maintaining identity consistency. Achieving this requires precise action control, spatial structure adaptation, and consistency preservation. To this end, we introduce RefAdapter, a lightweight image-conditioned adapter that excels in spatial adaptation and consistency preservation, surpassing existing methods in balancing appearance consistency and structural flexibility. Additionally, based on our observations, the denoising process exhibits varying levels of attention to motion (low frequency) and appearance details (high frequency) at different timesteps. So we propose FAE (Frequency-aware Action Extraction), which, unlike existing methods that rely on separate spatial-temporal architectures, directly achieves action extraction during the denoising process. Experiments demonstrate that our method effectively transfers actions to subjects with diverse layouts, skeletons, and viewpoints. We release our code and model weights to support further research at https://shiyi-zh0408.github.io/projectpages/FlexiAct/

Problem

• Transfer actions from reference video to target image
• Allow variations in layout, viewpoint, and skeletal structure
• Achieve precise action control and consistency preservation

Innovation

• FlexiAct transfers actions between diverse subjects and scenarios
• RefAdapter ensures spatial adaptation and identity consistency
• FAE achieves action extraction during denoising process