CDAE: Enhancing Perturbation Robustness in Pretrained Language Models with Contrastive Denoising

2026-07-30

AI Summary

This work addresses the limited robustness of sentence embeddings from pretrained language models, which are highly sensitive to semantic-preserving textual perturbations such as synonym substitution, masking, and word deletion. To mitigate this issue, the authors propose a lightweight Contrastive Denoising Autoencoder (CDAE), which uniquely integrates contrastive learning with denoising autoencoding. By jointly optimizing contrastive and reconstruction losses, CDAE learns sentence representations that are both invariant to perturbations and faithful to original semantics. The framework accommodates diverse perturbation strategies and consistently outperforms BERT and SimCSE across varying perturbation intensities, with performance gains becoming more pronounced under stronger perturbations. This demonstrates CDAE’s effectiveness in enhancing the stability and robustness of sentence embeddings.

Abstract

Pre-trained language models have significantly improved sentence representation learning, yet their embedding remain sensitive to semantic preserving textual perturbations such as synonym substitution, masking and word dropout. This work proposes a lightweight Contrastive Denoising Autoencoder (CDAE) that refines pre-trained BERT embedding by jointly optimizing contrastive and reconstruction objective to learn perturbation-invariant representation. We evaluate the proposed framework using multiple perturbation strategies with varying strengths and compare it against the original BERT embeddings and SimCSE. Experimental results show that CDAE consistently preserves higher embedding similarity under perturbations, with the improvements becoming more pronounced as framework effectively enhances representation stability while preserving semantic information, highlighting perturbation-invariant learning as a promising direction for improving sentence embeddings. The source code is publicly available at: https://github.com/ComputationIASBS/CDAE

Problem

• perturbation robustness
• pretrained language models
• sentence embeddings
• semantic preservation
• textual perturbations

Innovation

• Contrastive Denoising Autoencoder
• Perturbation Robustness
• Sentence Embeddings
• Pretrained Language Models
• Representation Stability