Institution profile

Dartmouth College

Academic institutionnorthamerica · us
Official website
Research library395linked papers
Opportunities0open roles
Selected work

Representative Papers

OmniBench: Towards The Future of Universal Omni-Language Models

Sep 23, 2024arXiv.org

Existing open-source multimodal large language models (MLLMs) exhibit significant deficiencies in joint visual-auditory-textual understanding and reasoning, achieving only ~50% instruction-following accuracy on trilingual multimodal tasks. Method: We introduce OmniBench—the first benchmark for trilingual multimodal collaborative reasoning—and formalize the omni-language model (OLM), a unified architecture capable of jointly processing visual, auditory, and textual (V-A-T) inputs. We construct OmniBench via expert human annotation across diverse trilingual multimodal tasks and curate OmniInstruct, a large-scale instruction-tuning dataset comprising 96K samples. Our methodology integrates cross-modal alignment modeling, trilingual multimodal instruction tuning, and a human-in-the-loop evaluation framework. Contribution/Results: Experiments reveal severe generalization limitations of current open-source OLMs on trilingual multimodal tasks; OmniInstruct substantially improves their reasoning performance. This work establishes a novel evaluation paradigm, provides high-quality resources, and outlines a technical pathway for advancing trilingual multimodal foundation models.

9 citations2 influentialRead paper

FLTrojan: Privacy Leakage Attacks against Federated Language Models Through Selective Weight Tampering

Oct 24, 2023arXiv.org

This work uncovers a novel targeted privacy leakage threat against federated language models (FLMs) in federated learning (FL): adversaries—via malicious clients—selectively perturb critical model weights during intermediate training rounds, enabling efficient extraction of others’ sensitive data without server cooperation. Unlike existing general-purpose data extraction attacks, we are the first to identify that intermediate model snapshots exhibit higher privacy vulnerability than final models, and we introduce a principled, weight-sensitivity-driven paradigm for targeted weight manipulation. Integrating membership inference, gradient analysis, and privacy-preserving data reconstruction modeling, our approach achieves a 29% improvement in membership inference recall and reconstructs private data with 71% accuracy—substantially outperforming strong-assumption baseline attacks.

7 citationsRead paper

Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Models

Jan 20, 2026

This work proposes a practical, three-stage “Locate–Guide–Improve” framework that transforms mechanistic interpretability from a post-hoc diagnostic tool into an engineering-driven optimization methodology for large language models. By systematically integrating techniques for identifying critical neurons and pathways with targeted interventions—such as activation manipulation and module editing—the framework establishes a standardized protocol for model refinement while clearly distinguishing between localization and guidance mechanisms. Empirical results demonstrate significant improvements in model alignment, task performance, and reasoning efficiency, thereby advancing mechanistic interpretability toward real-world applicability.

4 citationsRead paper

Addressing Overthinking in Large Vision-Language Models via Gated Perception-Reasoning Optimization

Jan 07, 2026arXiv.org

This work addresses the inefficiency and accuracy degradation of large vision-language models (VLMs) on simple tasks, where over-reasoning often leads to unnecessarily verbose responses. While prior approaches overlook visual perception failure as a fundamental bottleneck, this paper proposes GPRO, a novel framework that decouples perception failures from reasoning errors for the first time. GPRO constructs supervision signals based on failure attribution and introduces a meta-reasoning controller that dynamically selects among a lightweight fast path, a slow perception path, or a slow reasoning path. Leveraging a teacher model to generate approximately 790,000 failure-attribution labels, the path selection strategy is optimized via multi-objective reinforcement learning. Experiments demonstrate that GPRO significantly improves both accuracy and inference efficiency across five benchmarks, outperforming existing "slow thinking" methods while producing more concise responses.

3 citationsRead paper

Eigenspectrum Analysis of Neural Networks without Aspect Ratio Bias

Jun 06, 2025

This work addresses the bias in heavy-tailedness estimation arising from aspect-ratio disparities in weight matrices during spectral analysis of deep neural networks. We propose Fixed-Aspect-Ratio Matrix Sampling (FARMS), a method that mitigates this bias by randomly sampling submatrices with fixed aspect ratios, modeling their empirical spectral density (ESD), and fitting α-stable distributions to estimate the tail index. FARMS is the first framework to systematically eliminate the intrinsic aspect-ratio-induced bias in spectral statistics. It exhibits strong cross-architecture and cross-task robustness, significantly improving model diagnostics and layer-wise hyperparameter allocation. Extensive validation across computer vision (CV), scientific machine learning (SciML), and large language model (LLM) pruning tasks confirms its effectiveness: when applied to LLaMA-7B pruning, FARMS reduces perplexity by 17.3%, outperforming state-of-the-art methods.

2 citationsRead paper
Recent publications

Latest Papers

Design-Assisted Regression

Sep 11, 2026

本文提出一种设计辅助回归框架，通过利用协变量分布信息来稳定弱设计方向和修正潜在效应扭曲，从而改进估计性能。

0 citationsRead paper