Institution profile

FAR AI

Industry researchnorthamerica · us
Official website
Research library34linked papers
Opportunities0open roles
Selected work

Representative Papers

TamperBench: Systematically Stress-Testing LLM Safety Under Fine-Tuning and Tampering

Feb 06, 2026

This work addresses the absence of a unified benchmark for evaluating the safety robustness of large language models (LLMs) under fine-tuning and adversarial tampering, which hinders systematic comparison of their safety, utility, and resilience. We propose the first comprehensive and reproducible evaluation framework for LLM tamper resistance, integrating attacks in both weight space (e.g., jailbreak-tuning) and latent representation space, coupled with systematic hyperparameter sweeps and alignment-based defense mechanisms such as Triplet. The framework introduces standardized metrics for safety and capability assessment. Evaluations across 21 open-source LLMs under nine tampering threats reveal that jailbreak-tuning is the most destructive attack vector, Triplet emerges as the most effective defense, and post-training stages critically influence model robustness against tampering.

1 citationsRead paper

Scaling Trends for Lie Detector Oversight in Preference Learning

Jul 01, 2026

This work addresses the challenge of deceptive behavior in large language models during preference learning, where human annotation is costly. It presents the first application of the scalable oversight framework SOLiD to an ultra-large-scale model (405B parameters), demonstrating its effectiveness in more realistic and diverse preference learning settings. By integrating a high-precision lie detector achieving 99% true positive rate to filter suspicious responses, the approach substantially reduces the need for human review and eliminates human annotations entirely during fine-tuning, while limiting undetected deception to 14%. The study also reveals that the method is sensitive to distributional shifts between training and preference data, which can lead to elevated false positive rates.

0 citationsRead paper

Evaluating using Mock Tool Calls to Quarantine Untrusted Prompt Inputs

May 28, 2026

This work systematically investigates the vulnerability of large language models (LLMs) to prompt injection attacks when processing untrusted inputs and evaluates a novel defense strategy that encapsulates such inputs as simulated tool calls to leverage trust isolation mechanisms inherent in the model’s instruction hierarchy. Using an automated red-teaming framework, the authors assess this approach across seven prominent LLMs on three LLM-as-a-Judge tasks. Contrary to expectations, tool encapsulation does not consistently improve robustness; in binary judgment tasks such as GSM8K scoring, it significantly increases attack success rates. Moreover, certain models exhibit instruction hierarchy inversion, wherein higher-level directives are overridden by lower-level injected content. These findings reveal critical limitations and counterintuitive behaviors in current LLM architectures under real-world deployment scenarios.

0 citationsRead paper
Recent publications

Latest Papers

Scaling Trends for Lie Detector Oversight in Preference Learning

Jul 01, 2026

This work addresses the challenge of deceptive behavior in large language models during preference learning, where human annotation is costly. It presents the first application of the scalable oversight framework SOLiD to an ultra-large-scale model (405B parameters), demonstrating its effectiveness in more realistic and diverse preference learning settings. By integrating a high-precision lie detector achieving 99% true positive rate to filter suspicious responses, the approach substantially reduces the need for human review and eliminates human annotations entirely during fine-tuning, while limiting undetected deception to 14%. The study also reveals that the method is sensitive to distributional shifts between training and preference data, which can lead to elevated false positive rates.

0 citationsRead paper

Evaluating using Mock Tool Calls to Quarantine Untrusted Prompt Inputs

May 28, 2026

This work systematically investigates the vulnerability of large language models (LLMs) to prompt injection attacks when processing untrusted inputs and evaluates a novel defense strategy that encapsulates such inputs as simulated tool calls to leverage trust isolation mechanisms inherent in the model’s instruction hierarchy. Using an automated red-teaming framework, the authors assess this approach across seven prominent LLMs on three LLM-as-a-Judge tasks. Contrary to expectations, tool encapsulation does not consistently improve robustness; in binary judgment tasks such as GSM8K scoring, it significantly increases attack success rates. Moreover, certain models exhibit instruction hierarchy inversion, wherein higher-level directives are overridden by lower-level injected content. These findings reveal critical limitations and counterintuitive behaviors in current LLM architectures under real-world deployment scenarios.

0 citationsRead paper

Decoding Answers Before Chain-of-Thought: Evidence from Pre-CoT Probes and Activation Steering

Mar 01, 2026

This study investigates whether chain-of-thought (CoT) reasoning faithfully reflects the true decision-making process of large language models. By training linear probes on residual stream activations preceding CoT generation to predict final answers—and validating causal influence through activation interventions—the work provides the first mechanistic evidence that models typically commit to an answer before generating the CoT. Experimental results show that probes achieve AUC scores up to 0.9 across most tasks, and targeted activation steering flips the model’s answer in over 50% of samples, substantially outperforming baselines. Furthermore, the analysis reveals that when models hold incorrect beliefs, post-hoc CoT reasoning often leads to characteristic failure modes such as non-entailment or hallucination.

0 citationsRead paper