🤖 AI Summary
Current alignment methods for large language models exhibit limited rejection capability against emerging proxy attacks and incur high computational overhead. This work proposes RAG-Pref, a training-free online alignment approach that, for the first time, integrates retrieval-augmented generation (RAG) into the alignment pipeline. By dynamically retrieving preferred and non-preferred response pairs during inference and performing contrastive conditional reasoning, RAG-Pref substantially enhances the model’s ability to reject proxy attacks. Experimental results demonstrate that RAG-Pref improves the average rejection rate against proxy attacks by 3.7× across five mainstream large language models—outperforming offline alignment methods by 1.5×—while simultaneously strengthening general human preference alignment, all with negligible additional computational cost.
📝 Abstract
Large language model (LLM) alignment algorithms typically consist of post-training over preference pairs. While such algorithms are widely used to enable safety guardrails and align LLMs with general human preferences, we show that state-of-the-art alignment algorithms require significant computational resources while being far less capable of enabling refusal guardrails for recent agentic attacks. Thus, to improve refusal guardrails against such attacks without drastically increasing computational overhead, we introduce Retrieval Augmented Generation for Pref erence alignment (RAG-Pref), a simple RAG-based alignment algorithm which conditions on preferred and dispreferred samples to leverage contrastive information during inference. RAG-Pref is online (training-free), compatible with off-the-shelf packages, and, when combined with offline (training-based) alignment algorithms, enables more than an average 3.7 factor improvement in agentic attack refusals across five widely used LLMs, compared to 2.9 for other online alignment algorithms and 1.5 for offline alignment alone. We conclude by showing that, in stark contrast to other online alignment methods, RAG-Pref similarly increases performance on general human-preference alignment tasks and does not drastically increase overall computational requirements.