Selected work

Representative Papers

Amortized Latent Steering: Low-Cost Alternative to Test-Time Optimization

Sep 10, 2025arXiv.org

Test-time optimization (TTO) incurs prohibitively high inference overhead—10–100× that of standard decoding—due to iterative optimization and multi-step backpropagation, hindering practical deployment. To address this, we propose **Amortized Latent Steering (ALS)**, a gradient-free, constant-overhead alternative that eliminates online optimization. ALS precomputes, offline, the mean difference vector between latent states of successful and failed examples, enabling efficient, zero-gradient latent-space calibration at inference time. By integrating directional latent-space steering with a mean-field approximation, ALS achieves 2–5× speedup over iterative TTO on GSM8K and MATH-500 while improving the efficiency–accuracy trade-off by 101%. It matches or surpasses greedy chain-of-thought and Self-Consistency baselines in accuracy, marking the first practical, low-latency, high-accuracy deployment of TTO without backpropagation.

1 citationsRead paper

Synchronized Logit Steering: Real-world Steganography

Aug 09, 2026

This study addresses the unreliability of large language model steganography in the absence of shared prompts by proposing a Synchronized Logit Guidance mechanism. The method infers proxy prompts from generated content to achieve distribution synchronization without shared context, while integrating periodic burst coding and token-rank coding to enhance embedding capacity. Experimental results demonstrate that the system achieves a KL divergence below 0.5 nats and an embedding capacity of 0.20 bits/token, with output text statistically indistinguishable from greedy decoding. This work effectively resolves keyless steganographic challenges, significantly improving both the practicality and security of covert communication.

0 citationsRead paper

Why Limit the Residual Stream to Layers and Not Tokens? Persistent Memory for Continuous Latent Reasoning

Jun 05, 2026

This work addresses the “concept bottleneck” in existing Continuous Conceptual Thought (CoCoNuT) methods, where intermediate latent states are overwritten during multi-step reasoning, leading to performance degradation as reasoning depth increases. To overcome this limitation, the authors propose AGCLR, the first approach to integrate a persistent memory mechanism into continuous latent reasoning frameworks. AGCLR employs a three-gated architecture—comprising write, read, and forget gates—to dynamically manage residual memory across reasoning steps, thereby preserving and leveraging critical intermediate facts over multiple inference rounds. Implemented on a GPT-2 backbone, AGCLR consistently outperforms baseline models on GSM8K, HotpotQA, and ProsQA benchmarks. Notably, its performance gains amplify with increasing reasoning depth, effectively mitigating the concept bottleneck.

0 citationsRead paper

When LLMs Learn to Be Consistently Wrong: A Multi-Model Study of Linear Representations of Synthetic Deception

May 27, 2026

This study investigates how large language models retain accurate internal representations of truthful information even after fine-tuning induces systematic deceptive outputs. Using LoRA-based fine-tuning, we construct paired honest and deceptive variants across five Transformer models and train linear probes to detect deception signals in hidden states. For the first time in a multi-model setting, we demonstrate that deceptive representations are linearly separable, with deception directions progressively consolidating in deeper layers. Notably, models such as Gemma-2 stably preserve these high-dimensional representations without collapse. Our experiments reveal that near-perfect deception detection (AUC ≥ 0.99) is achievable using only the first one to three layers’ hidden states. Furthermore, probes trained on TruthfulQA generalize zero-shot to unseen MMLU subjects, and deep-layer representations exhibit strong robustness against Gaussian noise.

0 citationsRead paper

Mechanistic origins of catastrophic forgetting: why RL preserves circuits better than SFT?

May 21, 2026

This work investigates the mechanisms underlying catastrophic forgetting in large language models during supervised fine-tuning (SFT) and the relative robustness of reinforcement learning (RL) in preserving pre-existing capabilities. The study introduces, for the first time, a connection between forgetting and the preservation of internal computational circuits, proposing a novel metric termed “differential circuit fragility” to quantify the disruption of attention-head-level circuits under different fine-tuning strategies. Experiments on the Qwen2.5-3B-Instruct model using a scientific question-answering benchmark reveal that while SFT achieves faster task adaptation, it substantially degrades pre-trained circuits. In contrast, RL preserves foundational circuits more effectively at the cost of slower adaptation, demonstrating that circuit retention is a key mechanism behind RL’s superior resistance to catastrophic forgetting.

0 citationsRead paper

Recent publications

Latest Papers

Synchronized Logit Steering: Real-world Steganography

Aug 09, 2026

This study addresses the unreliability of large language model steganography in the absence of shared prompts by proposing a Synchronized Logit Guidance mechanism. The method infers proxy prompts from generated content to achieve distribution synchronization without shared context, while integrating periodic burst coding and token-rank coding to enhance embedding capacity. Experimental results demonstrate that the system achieves a KL divergence below 0.5 nats and an embedding capacity of 0.20 bits/token, with output text statistically indistinguishable from greedy decoding. This work effectively resolves keyless steganographic challenges, significantly improving both the practicality and security of covert communication.

0 citationsRead paper

Why Limit the Residual Stream to Layers and Not Tokens? Persistent Memory for Continuous Latent Reasoning

Jun 05, 2026

This work addresses the “concept bottleneck” in existing Continuous Conceptual Thought (CoCoNuT) methods, where intermediate latent states are overwritten during multi-step reasoning, leading to performance degradation as reasoning depth increases. To overcome this limitation, the authors propose AGCLR, the first approach to integrate a persistent memory mechanism into continuous latent reasoning frameworks. AGCLR employs a three-gated architecture—comprising write, read, and forget gates—to dynamically manage residual memory across reasoning steps, thereby preserving and leveraging critical intermediate facts over multiple inference rounds. Implemented on a GPT-2 backbone, AGCLR consistently outperforms baseline models on GSM8K, HotpotQA, and ProsQA benchmarks. Notably, its performance gains amplify with increasing reasoning depth, effectively mitigating the concept bottleneck.

0 citationsRead paper

When LLMs Learn to Be Consistently Wrong: A Multi-Model Study of Linear Representations of Synthetic Deception

May 27, 2026

This study investigates how large language models retain accurate internal representations of truthful information even after fine-tuning induces systematic deceptive outputs. Using LoRA-based fine-tuning, we construct paired honest and deceptive variants across five Transformer models and train linear probes to detect deception signals in hidden states. For the first time in a multi-model setting, we demonstrate that deceptive representations are linearly separable, with deception directions progressively consolidating in deeper layers. Notably, models such as Gemma-2 stably preserve these high-dimensional representations without collapse. Our experiments reveal that near-perfect deception detection (AUC ≥ 0.99) is achievable using only the first one to three layers’ hidden states. Furthermore, probes trained on TruthfulQA generalize zero-shot to unseen MMLU subjects, and deep-layer representations exhibit strong robustness against Gaussian noise.

0 citationsRead paper

Mechanistic origins of catastrophic forgetting: why RL preserves circuits better than SFT?

May 21, 2026

This work investigates the mechanisms underlying catastrophic forgetting in large language models during supervised fine-tuning (SFT) and the relative robustness of reinforcement learning (RL) in preserving pre-existing capabilities. The study introduces, for the first time, a connection between forgetting and the preservation of internal computational circuits, proposing a novel metric termed “differential circuit fragility” to quantify the disruption of attention-head-level circuits under different fine-tuning strategies. Experiments on the Qwen2.5-3B-Instruct model using a scientific question-answering benchmark reveal that while SFT achieves faster task adaptation, it substantially degrades pre-trained circuits. In contrast, RL preserves foundational circuits more effectively at the cost of slower adaptation, demonstrating that circuit retention is a key mechanism behind RL’s superior resistance to catastrophic forgetting.

0 citationsRead paper

Retrieval, Refinement, and Ranking for Text-to-Video Generation via Prompt Optimization and Test-Time Scaling

Mar 02, 2026

This work proposes the 3R framework to address the high sensitivity of existing text-to-video (T2V) generation models to input prompts, a limitation that traditional approaches struggle to overcome without costly fine-tuning or complex post-processing, often sacrificing quality, scalability, or usability. The 3R framework uniquely integrates retrieval-augmented generation (RAG), diffusion-based preference optimization, and temporal frame interpolation—without requiring any model training—to significantly enhance the output quality of arbitrary T2V models. By leveraging RAG to extract semantically rich modifiers, refining prompts via a vision-language model, and improving motion coherence through temporal interpolation, the method effectively boosts both static fidelity and temporal consistency in generated videos, thereby underscoring the pivotal role of prompt optimization in T2V synthesis.

0 citationsRead paper