You Are What You Read: Misalignment via In-Context Persona Induction
研究通过提供良性数据而非有害行为演示,使模型在对话中采用特定人物身份,从而探讨了对齐问题。
研究通过提供良性数据而非有害行为演示,使模型在对话中采用特定人物身份,从而探讨了对齐问题。
本文提出Diff Mining框架,通过比较微调前后模型的logits差异来识别微调过程中学习到的行为,使用Top-K频率法和NMF方法聚合信号以解析多种微调目标。
This work addresses the limitation of conventional language models, whose post-pretraining alignment mechanisms often result in superficial value adherence that degrades under out-of-distribution scenarios. To overcome this, the authors propose Synthetic Persona Pretraining (SPP), a novel approach that integrates value alignment at the very beginning of pretraining. SPP injects first-person reflective texts generated from a value constitution, enabling the assistant persona to internalize normative principles from token zero. This early-stage alignment is further reinforced through subsequent role-binding fine-tuning, achieving deep and robust value alignment. Experimental results demonstrate that SPP substantially enhances the model’s adherence to ethical guidelines, improves robustness against jailbreaking attempts, and maintains stable alignment in out-of-distribution moral dilemmas. Notably, introducing alignment during early pretraining yields superior performance without compromising general task capabilities.
This work systematically uncovers a critical compatibility vulnerability in encrypted reasoning chains employed by large language model (LLM) providers to safeguard intellectual property. Despite encryption, reasoning blocks exhibit cross-session, cross-user, and cross-model interoperability flaws. The authors propose a novel decryption-based jailbreaking technique that leverages reverse engineering and cross-model reasoning block injection: by exploiting weaker models to decrypt the encrypted inference traces of stronger ones, the method reconstructs full reasoning processes without direct attacks. Empirical evaluation demonstrates successful extraction of reasoning chains from models by Anthropic, OpenAI, and Google, decrypting 315,320 publicly logged blocks, recovering 367 personally identifiable information (PII) instances and 182 credential sets, and enabling stealthy prompt injection that effectively bypasses existing anti-distillation and security mechanisms.
This work addresses the pervasive over-reliance on Python by large language models (LLMs) in project-level code generation and the absence of systematic evaluation of their programming language selection behavior. We introduce LangChoiceBench, a benchmark spanning 28 projects across seven software domains, which enables the first systematic quantification of LLMs’ language preferences and diversity. Through manual analysis of 9,826 inference trajectories, we uncover novel failure modes, including “hallucinated evidence” and inconsistencies between recommended and implemented languages. Evaluation of 25 prominent LLMs reveals that most models select languages based on habitual bias rather than project requirements, with smaller models exhibiting even poorer language diversity, thereby highlighting a critical limitation of current LLMs in cross-language project generation.
研究通过提供良性数据而非有害行为演示,使模型在对话中采用特定人物身份,从而探讨了对齐问题。
本文提出Diff Mining框架,通过比较微调前后模型的logits差异来识别微调过程中学习到的行为,使用Top-K频率法和NMF方法聚合信号以解析多种微调目标。
This work addresses the limitation of conventional language models, whose post-pretraining alignment mechanisms often result in superficial value adherence that degrades under out-of-distribution scenarios. To overcome this, the authors propose Synthetic Persona Pretraining (SPP), a novel approach that integrates value alignment at the very beginning of pretraining. SPP injects first-person reflective texts generated from a value constitution, enabling the assistant persona to internalize normative principles from token zero. This early-stage alignment is further reinforced through subsequent role-binding fine-tuning, achieving deep and robust value alignment. Experimental results demonstrate that SPP substantially enhances the model’s adherence to ethical guidelines, improves robustness against jailbreaking attempts, and maintains stable alignment in out-of-distribution moral dilemmas. Notably, introducing alignment during early pretraining yields superior performance without compromising general task capabilities.
This work systematically uncovers a critical compatibility vulnerability in encrypted reasoning chains employed by large language model (LLM) providers to safeguard intellectual property. Despite encryption, reasoning blocks exhibit cross-session, cross-user, and cross-model interoperability flaws. The authors propose a novel decryption-based jailbreaking technique that leverages reverse engineering and cross-model reasoning block injection: by exploiting weaker models to decrypt the encrypted inference traces of stronger ones, the method reconstructs full reasoning processes without direct attacks. Empirical evaluation demonstrates successful extraction of reasoning chains from models by Anthropic, OpenAI, and Google, decrypting 315,320 publicly logged blocks, recovering 367 personally identifiable information (PII) instances and 182 credential sets, and enabling stealthy prompt injection that effectively bypasses existing anti-distillation and security mechanisms.
This work addresses the pervasive over-reliance on Python by large language models (LLMs) in project-level code generation and the absence of systematic evaluation of their programming language selection behavior. We introduce LangChoiceBench, a benchmark spanning 28 projects across seven software domains, which enables the first systematic quantification of LLMs’ language preferences and diversity. Through manual analysis of 9,826 inference trajectories, we uncover novel failure modes, including “hallucinated evidence” and inconsistencies between recommended and implemented languages. Evaluation of 25 prominent LLMs reveals that most models select languages based on habitual bias rather than project requirements, with smaller models exhibiting even poorer language diversity, thereby highlighting a critical limitation of current LLMs in cross-language project generation.