Selected work

Representative Papers

AdvPrefix: An Objective for Nuanced LLM Jailbreaks

Dec 13, 2024arXiv.org

Existing LLM jailbreaking attacks suffer from weak controllability, incomplete response generation, and rigid, non-adaptive optimization formats. To address these limitations, we propose AdvPrefix—a novel prefix-based objective function designed for fine-grained jailbreaking. AdvPrefix introduces the first model-adaptive prefix selection mechanism, which automatically identifies high-quality prefixes during the prefilling stage using a dual criterion: attack success rate and negative log-likelihood. It further enables multi-prefix collaborative optimization, departing from conventional fixed-prefix paradigms and exposing alignment models’ generalization vulnerabilities to unseen prefixes. AdvPrefix is fully compatible with mainstream optimization frameworks (e.g., GCG) and requires no model modification or retraining. Evaluated on Llama-3, AdvPrefix boosts GCG’s fine-grained jailbreaking success rate from 14% to 80%, demonstrating the critical impact of objective function design on jailbreaking efficacy.

10 citations3 influentialRead paper

On the origin of neural scaling laws: from random graphs to natural language

Jan 15, 2026

This study investigates whether neural scaling laws depend on power-law structures in data. By training a minimal Transformer—comprising only two layers and a context length of 50—to predict random walk sequences on synthetic graphs of tunable complexity (including Erdős–Rényi and Barabási–Albert models), the authors systematically simplify the language modeling task. They demonstrate, for the first time, that neural scaling laws emerge even in data devoid of power-law correlations, revealing a monotonic relationship between linguistic complexity and the scaling exponent. This finding establishes that power-law structures in data are not necessary for the emergence of scaling laws. Furthermore, the work introduces an alternative method for constructing compute-optimal curves and shows that the Maximal Update Parametrization (μP) achieves superior parameter efficiency.

3 citationsRead paper

Memorization Dynamics in Knowledge Distillation for Language Models

Jan 21, 2026

This study systematically investigates the dynamics of training data memorization in knowledge distillation for language models and its implications for privacy and generalization. The authors demonstrate that distilled models exhibit over 50% lower memorization rates compared to standard fine-tuned counterparts, with more than 95% of memorized content attributable to a small subset of highly memorable samples. Through experiments across Pythia, OLMo-2, and Qwen-3 models trained on FineWeb, Wikitext, and Nemotron-CC-v2 datasets, they reveal that hard distillation incurs 2.7 times higher risk of inheriting teacher-specific memorized samples than soft distillation. Leveraging metrics such as zlib entropy, KL divergence, and perplexity, the work further introduces a prior predictability method to anticipate memorization propensity. These findings indicate that knowledge distillation not only enhances generalization but also significantly mitigates privacy risks associated with data memorization.

1 citationsRead paper

Measuring Dejavu Memorization Efficiently

Apr 08, 2025Neural Information Processing Systems

Existing methods for evaluating memorization in representation learning models—such as the DejaVu phenomenon—require training multiple models, rendering them infeasible for large-scale open-source models. Method: We propose a lightweight, zero-shot, retraining-free evaluation framework that models data-level correlations via single-model feature statistics and integrates a background–foreground memory estimation protocol inspired by the DejaVu paradigm. Contribution/Results: Our method enables the first efficient, scalable, and architecture-agnostic quantification of memorization in mainstream open-source vision and multimodal representation models—including CLIP and DINOv2. Extensive experiments demonstrate high consistency across diverse metrics and reveal that large open-source models exhibit significantly lower overall memorization than comparably sized models trained on subsets of the same data. This breakthrough overcomes the long-standing scalability bottleneck in memorization assessment for large foundation models.

1 citationsRead paper

DecompRL: Solving Harder Problems by Learning Modular Code Generation

Jul 02, 2026

This work addresses the challenge that large language models struggle to solve extremely difficult programming problems due to the near-zero probability of sampling correct solutions through conventional methods or reinforcement learning (RL). To overcome this, we propose DecompRL, the first RL approach tailored for modular code generation. DecompRL decomposes complex problems into independently solvable subfunctions, learns to generate composable hierarchical code structures, and leverages verifiable reward signals for training. This framework enables exponential combinatorial exploration of candidate solutions while allowing efficient CPU-based evaluation of composed programs, drastically reducing GPU inference costs. Experiments on LiveCodeBench and CodeContests demonstrate that DecompRL substantially outperforms both standard and diversity-enhanced RL baselines, successfully solving problems intractable to prior methods while reducing GPU token consumption by approximately 50×.

0 citationsRead paper

Recent publications

Latest Papers

DecompRL: Solving Harder Problems by Learning Modular Code Generation

Jul 02, 2026

This work addresses the challenge that large language models struggle to solve extremely difficult programming problems due to the near-zero probability of sampling correct solutions through conventional methods or reinforcement learning (RL). To overcome this, we propose DecompRL, the first RL approach tailored for modular code generation. DecompRL decomposes complex problems into independently solvable subfunctions, learns to generate composable hierarchical code structures, and leverages verifiable reward signals for training. This framework enables exponential combinatorial exploration of candidate solutions while allowing efficient CPU-based evaluation of composed programs, drastically reducing GPU inference costs. Experiments on LiveCodeBench and CodeContests demonstrate that DecompRL substantially outperforms both standard and diversity-enhanced RL baselines, successfully solving problems intractable to prior methods while reducing GPU token consumption by approximately 50×.

0 citationsRead paper

When to Align, When to Predict: A Phase Diagram for Multimodal Learning

Jun 09, 2026

This work addresses the lack of a systematic understanding of when cross-modal alignment (CA) and cross-modal prediction (CP) are effective in multimodal learning—a gap that often leads to suboptimal performance or even degradation relative to unimodal baselines. The authors propose a unified linear analytical framework under a structured signal–noise model with correlated interference, revealing complementary failure mechanisms of CA and CP. They introduce the first multimodal “phase diagram,” which delineates four distinct regimes: both methods succeed, only alignment works, only prediction works, or neither is effective. Leveraging separation ratio analysis, a unidirectional whitening mechanism, and a few-shot label-guided localization algorithm, this phase diagram enables practical guidance for method selection on real-world data. Experiments across synthetic, stereo vision, image–text, and astrophysical datasets validate its efficacy in identifying harmful multimodal configurations, offering a diagnostic tool for practitioners prior to model deployment.

0 citationsRead paper

TextSeal: A Localized LLM Watermark for Provenance & Distillation Protection

May 12, 2026

This work addresses the challenge of tracing content generated by large language models, where existing watermarking methods suffer from limitations in diversity, robustness, and protection against distillation. The authors propose a theoretically distortion-free, localizable watermarking mechanism based on Gumbel-max sampling. By employing dual keys to restore output diversity and integrating entropy-weighted scoring with multi-region localization, the method enhances detection capability while remaining compatible with efficient inference techniques such as speculative decoding—achieving zero inference overhead. Notably, it is the first scheme to enable watermark transfer after model distillation (“radioactive” property) and to precisely localize AI-generated segments within human-AI hybrid text. Experiments demonstrate significantly stronger detection performance than baselines like SynthID-text and robustness against dilution attacks. Multilingual human evaluations (five languages, 6,000 A/B tests) confirm imperceptible quality degradation and no adverse impact on downstream task performance.

0 citationsRead paper

A single algorithm for both restless and rested rotting bandits

Apr 23, 2026

This work addresses the multi-armed bandit problem under time-decaying rewards, unifying the treatment of both “rested” (history-dependent) and “restless” (externally driven) rotting settings. The paper proposes the Rotting Adaptive Window UCB (RAW-UCB) algorithm, which achieves near-optimal regret bounds for both rotting bandit variants within a single framework—without requiring prior knowledge of the environment type or non-stationarity pattern. RAW-UCB integrates an adaptive sliding window with the upper confidence bound (UCB) principle and incorporates a robust model of reward decay dynamics. Theoretical analysis establishes its near-optimality, while experiments on both synthetic and real-world datasets demonstrate its effectiveness and strong generalization capability across diverse rotting scenarios.

0 citationsRead paper

AA-SVD : Anchored and Adaptive SVD for Large Language Model Compression

Apr 02, 2026

Existing SVD-based compression methods for large language models struggle to simultaneously preserve the fidelity of original outputs and account for input distribution shifts induced by upstream compression, often leading to error accumulation and performance degradation. This work proposes a fast, training-free low-rank compression framework that, within a single-layer decomposition, jointly anchors the original output and models input distribution shifts. By end-to-end optimizing entire Transformer blocks to minimize output distortion, the method innovatively integrates output anchoring, adaptive input modeling, and block-level joint optimization. It consistently outperforms existing SVD-based baselines across various compression ratios and maintains stable performance even under high compression, effectively avoiding the catastrophic performance collapse observed in other approaches.

0 citationsRead paper