Selected work

Representative Papers

OmniBench: Towards The Future of Universal Omni-Language Models

Sep 23, 2024arXiv.org

Existing open-source multimodal large language models (MLLMs) exhibit significant deficiencies in joint visual-auditory-textual understanding and reasoning, achieving only ~50% instruction-following accuracy on trilingual multimodal tasks. Method: We introduce OmniBench—the first benchmark for trilingual multimodal collaborative reasoning—and formalize the omni-language model (OLM), a unified architecture capable of jointly processing visual, auditory, and textual (V-A-T) inputs. We construct OmniBench via expert human annotation across diverse trilingual multimodal tasks and curate OmniInstruct, a large-scale instruction-tuning dataset comprising 96K samples. Our methodology integrates cross-modal alignment modeling, trilingual multimodal instruction tuning, and a human-in-the-loop evaluation framework. Contribution/Results: Experiments reveal severe generalization limitations of current open-source OLMs on trilingual multimodal tasks; OmniInstruct substantially improves their reasoning performance. This work establishes a novel evaluation paradigm, provides high-quality resources, and outlines a technical pathway for advancing trilingual multimodal foundation models.

9 citations2 influentialRead paper

SafeSearch: Automated Red-Teaming for the Safety of LLM-Based Search Agents

Sep 28, 2025

Large language model (LLM)-based search agents face significant security risks—including exposure to low-quality or malicious content—when interfacing with the open web. Method: We propose the first systematic, scalable, and cost-effective automated red-teaming framework for evaluating such agents. Our approach introduces SafeSearch, a dedicated safety benchmark comprising 300 test cases spanning five critical risk categories, and integrates automated attack generation to assess the security robustness of diverse search agent architectures—including search workflows, tool-augmented agents, and deep-research systems—across leading closed- and open-weight LLMs. Contribution/Results: We expose severe limitations of common mitigation strategies (e.g., prompt-based warnings); empirically demonstrate that GPT-4.1-mini achieves a 90.5% attack success rate against unreliable websites within search workflows; and advance standardization and transparency in search-agent safety evaluation, providing an empirical foundation for developing effective defenses.

0 citationsRead paper

Human Motion Video Generation: A Survey

Sep 04, 2025

Existing surveys predominantly focus on isolated methodologies, lacking a systematic examination of the end-to-end pipeline for human motion video generation. To address this gap, we propose the first unified framework encompassing five core stages: input specification, motion planning, generation, optimization, and output rendering—supporting over ten subtasks driven by visual, textual, and audio modalities. Our work systematically reviews 200+ papers and constructs the field’s first comprehensive technical taxonomy. We innovatively investigate the potential of large language models (LLMs) for motion semantic modeling and cross-modal alignment. Furthermore, we integrate state-of-the-art techniques—including diffusion models, generative adversarial networks (GANs), and multimodal fusion—to identify key breakthroughs and release an open-source model library. This study fills a critical void in holistic, cross-cutting research on human motion video generation, providing both theoretical foundations and practical guidelines for applications such as digital avatars.

0 citationsRead paper

OS Agents: A Survey on MLLM-based Agents for General Computing Devices Use

Aug 06, 2025

This paper introduces and formally defines the “Operating System Agent” (OS Agent) paradigm: a multimodal large language model (MLLM)-driven agent capable of autonomously executing cross-device, cross-application tasks within GUI-based operating system interfaces. Methodologically, it integrates GUI visual understanding, action-space modeling, hierarchical task planning, and human–agent collaborative interaction into a unified technical framework. Key contributions include: (1) the first systematic conceptualization and delineation of OS Agent capabilities and scope; (2) the establishment of the first fine-grained taxonomy and dedicated evaluation benchmark for OS Agents; (3) the open-sourcing of a dynamic resource repository—comprising annotated data, pretrained models, and modular toolchains—to support reproducible research and development; and (4) a comprehensive survey paper accepted at ACL 2025, providing both theoretical foundations and practical guidelines for academic advancement and industrial deployment.

0 citationsRead paper

Stepwise Schema-Guided Prompting Framework with Parameter Efficient Instruction Tuning for Multimedia Event Extraction

Jun 30, 2025IEEE International Conference on Multimedia and Expo

Multimodal Event Extraction (MEE) faces two key challenges: complex structural modeling and severe scarcity of fine-grained multimodal alignment annotations. To address these, we propose a stepwise pattern-guided prompting framework that decouples event type identification and argument role filling into two sequential prompting stages, and introduces text-bridged localization to achieve cross-modal fine-grained alignment. To alleviate the data bottleneck, we construct the first weakly aligned multimodal event dataset. Building upon the LLaVA-v1.5-7B foundation model, we employ LoRA for parameter-efficient instruction tuning, integrating multi-step prompting with bridged grounding. On the M2E2 benchmark, our method achieves +5.8 points in event detection F1 and +8.4 points in argument extraction F1 over prior state-of-the-art methods, demonstrating substantial improvements in both accuracy and robustness.

0 citationsRead paper

Recent publications

Latest Papers

SafeSearch: Automated Red-Teaming for the Safety of LLM-Based Search Agents

Sep 28, 2025

Large language model (LLM)-based search agents face significant security risks—including exposure to low-quality or malicious content—when interfacing with the open web. Method: We propose the first systematic, scalable, and cost-effective automated red-teaming framework for evaluating such agents. Our approach introduces SafeSearch, a dedicated safety benchmark comprising 300 test cases spanning five critical risk categories, and integrates automated attack generation to assess the security robustness of diverse search agent architectures—including search workflows, tool-augmented agents, and deep-research systems—across leading closed- and open-weight LLMs. Contribution/Results: We expose severe limitations of common mitigation strategies (e.g., prompt-based warnings); empirically demonstrate that GPT-4.1-mini achieves a 90.5% attack success rate against unreliable websites within search workflows; and advance standardization and transparency in search-agent safety evaluation, providing an empirical foundation for developing effective defenses.

0 citationsRead paper

Human Motion Video Generation: A Survey

Sep 04, 2025

Existing surveys predominantly focus on isolated methodologies, lacking a systematic examination of the end-to-end pipeline for human motion video generation. To address this gap, we propose the first unified framework encompassing five core stages: input specification, motion planning, generation, optimization, and output rendering—supporting over ten subtasks driven by visual, textual, and audio modalities. Our work systematically reviews 200+ papers and constructs the field’s first comprehensive technical taxonomy. We innovatively investigate the potential of large language models (LLMs) for motion semantic modeling and cross-modal alignment. Furthermore, we integrate state-of-the-art techniques—including diffusion models, generative adversarial networks (GANs), and multimodal fusion—to identify key breakthroughs and release an open-source model library. This study fills a critical void in holistic, cross-cutting research on human motion video generation, providing both theoretical foundations and practical guidelines for applications such as digital avatars.

0 citationsRead paper

OS Agents: A Survey on MLLM-based Agents for General Computing Devices Use

Aug 06, 2025

This paper introduces and formally defines the “Operating System Agent” (OS Agent) paradigm: a multimodal large language model (MLLM)-driven agent capable of autonomously executing cross-device, cross-application tasks within GUI-based operating system interfaces. Methodologically, it integrates GUI visual understanding, action-space modeling, hierarchical task planning, and human–agent collaborative interaction into a unified technical framework. Key contributions include: (1) the first systematic conceptualization and delineation of OS Agent capabilities and scope; (2) the establishment of the first fine-grained taxonomy and dedicated evaluation benchmark for OS Agents; (3) the open-sourcing of a dynamic resource repository—comprising annotated data, pretrained models, and modular toolchains—to support reproducible research and development; and (4) a comprehensive survey paper accepted at ACL 2025, providing both theoretical foundations and practical guidelines for academic advancement and industrial deployment.

0 citationsRead paper

Stepwise Schema-Guided Prompting Framework with Parameter Efficient Instruction Tuning for Multimedia Event Extraction

Jun 30, 2025IEEE International Conference on Multimedia and Expo

Multimodal Event Extraction (MEE) faces two key challenges: complex structural modeling and severe scarcity of fine-grained multimodal alignment annotations. To address these, we propose a stepwise pattern-guided prompting framework that decouples event type identification and argument role filling into two sequential prompting stages, and introduces text-bridged localization to achieve cross-modal fine-grained alignment. To alleviate the data bottleneck, we construct the first weakly aligned multimodal event dataset. Building upon the LLaVA-v1.5-7B foundation model, we employ LoRA for parameter-efficient instruction tuning, integrating multi-step prompting with bridged grounding. On the M2E2 benchmark, our method achieves +5.8 points in event detection F1 and +8.4 points in argument extraction F1 over prior state-of-the-art methods, demonstrating substantial improvements in both accuracy and robustness.

0 citationsRead paper

Vamba: Understanding Hour-Long Videos with Hybrid Mamba-Transformers

Mar 14, 2025

Transformer-based models face prohibitive computational overhead in hour-long video understanding due to quadratic complexity in sequence length, while existing token compression methods sacrifice spatiotemporal fidelity and still struggle with scalability. Method: We propose a novel hybrid Mamba–Transformer architecture that eliminates token compression entirely. It innovatively integrates the linear-complexity Mamba-2 state space model with Transformer layers via a hybrid attention mechanism, enabling efficient encoding of >1024 high-resolution frames at native spatiotemporal resolution. Coupled with an end-to-end multimodal encoder and long-sequence optimization strategies, it supports hour-long video encoding on a single GPU. Results: Our method reduces training/inference memory consumption by 50% and accelerates per-step throughput by nearly 2×. On LVBench, it achieves a 4.3% accuracy gain over prior state-of-the-art efficient video LLMs, while maintaining strong generalization across both long- and short-video understanding tasks.

0 citationsRead paper