Institution profile

Hong Kong University of Science and Technology

Academic institutionasia · hk
Official website
Research library4,954linked papers
Opportunities0open roles
Selected work

Representative Papers

TPTU: Large Language Model-based AI Agents for Task Planning and Tool Usage

Aug 07, 2023

To address LLMs’ weak task planning capability and brittle tool invocation in complex scenarios, this paper proposes TPTU—the first structured LLM agent framework that explicitly decouples and formalizes task planning and tool utilization as dual core competencies. It introduces a collaborative reasoning mechanism between single-step and sequential agents, supports extensible agent-type specialization, and integrates prompt engineering, dynamic tool selection, multi-step reasoning scheduling, and structured output parsing—ensuring compatibility with diverse mainstream LLMs. A systematic evaluation of 12 LLMs across representative tasks reveals three fundamental bottlenecks: insufficient planning depth, poor tool generalization, and weak error recovery. Based on these findings, we establish the first benchmark suite tailored for practical AI agents, providing both theoretical foundations and empirical pathways for advancing agent architecture design and capability enhancement. (149 words)

49 citations1 influentialRead paper

IdealGPT: Iteratively Decomposing Vision and Language Reasoning via Large Language Models

May 24, 2023Conference on Empirical Methods in Natural Language Processing

Existing vision-language models (VLMs) exhibit limited performance on zero-shot multi-step reasoning tasks, primarily due to their reliance on domain-specific subproblem decomposers and their tendency to force final answers even under insufficient information—compromising reasoning reliability. This paper proposes the first domain-agnostic, adaptive iterative decomposition framework: an LLM first generates subquestions; a VLM then provides visually grounded subanswers via multimodal grounding; finally, the LLM aggregates results and dynamically decides whether to terminate. This enables trustworthy, self-correcting convergence. The framework integrates zero-shot prompting synergy with a divide-and-conquer architecture, substantially enhancing reasoning robustness. Under zero-shot settings, it achieves absolute accuracy gains of +10.2% and +15.6% over the strongest GPT-4–based baselines on the VCR and SNLI-VE benchmarks, respectively.

42 citations6 influentialRead paper

Compositional Feature Augmentation for Unbiased Scene Graph Generation

Aug 13, 2023IEEE International Conference on Computer Vision

Scene Graph Generation (SGG) suffers from severe predicate long-tail distribution, where conventional re-sampling–based debiasing methods fail to improve tail-predicate performance—primarily due to insufficient modeling of relational triplet feature diversity. This paper introduces, for the first time, a feature disentanglement perspective: it decomposes triplet representations into intrinsic (predicate semantics) and extrinsic (contextual dependency) components. Building upon this, we propose a plug-and-play replace-mix augmentation strategy that enhances tail-predicate feature diversity without modifying the backbone architecture. The method is model-agnostic and computationally efficient. Evaluated on Visual Genome (VG) and PIC benchmarks, it achieves state-of-the-art performance across multiple metrics, notably boosting tail-predicate Recall@100 by a significant margin. Moreover, it seamlessly integrates with diverse SGG frameworks, demonstrating broad compatibility and practical utility.

27 citations4 influentialRead paper

SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild

Mar 24, 2025arXiv.org

This work investigates the universality and training dynamics of zero-shot reinforcement learning (Zero RL) across heterogeneous foundation models. Method: We systematically evaluate whether chain-of-thought (CoT) reasoning emerges directly from base models—without explicit CoT supervision—across ten open-source models spanning diverse architectures and scales. We introduce two key design principles: format reward shaping and query difficulty control, and integrate rule-based RL, implicit CoT supervision, joint monitoring of response length and verification behavior, and a cross-model training dynamics analysis framework. Contribution/Results: We observe, for the first time, a “reasoning insight moment” in non-Qwen small-scale models. Our analysis reveals a non-monotonic relationship between model scale and training dynamics. Experiments demonstrate significant improvements in reasoning accuracy and response length across most models. To foster reproducibility, we open-source all code, fine-tuned models, and analytical tools.

25 citations4 influentialRead paper

AtomThink: A Slow Thinking Framework for Multimodal Mathematical Reasoning

Nov 18, 2024arXiv.org

Multimodal large language models (MLLMs) exhibit limited capability in solving complex mathematical reasoning problems due to insufficient granular, stepwise reasoning over multimodal inputs. Method: This paper introduces the “slow-thinking” paradigm, integrating long-chain, atomic-level reasoning into MLLMs via AtomThink—a novel atomic thinking framework comprising (i) an automatic Chain-of-Thought (CoT) annotation engine, (ii) atomic-step fine-tuning, and (iii) a policy-based search method guided by a four-category strategy reward model (PRM). The approach unifies vision–math joint fine-tuning, reinforcement learning–driven search, and interpretable CoT generation. Contribution/Results: We release AtomMATH, a large-scale multimodal mathematical dataset, and propose fine-grained atomic capability evaluation metrics. On MathVista and MathVerse benchmarks, our method achieves relative accuracy improvements of ~50% and ~120%, respectively, significantly enhancing MLLMs’ hierarchical, adaptive reasoning on complex mathematical problems.

18 citations3 influentialRead paper
Recent publications

Latest Papers