Domain Lexical Knowledge-based Word Embedding Learning for Text Classification under Small Data

2025-06-02

AI Summary

In few-shot text classification, BERT embeddings often exhibit insufficient discriminative power—particularly when category predictions rely heavily on keywords. To address this, we propose a knowledge-guided BERT embedding enhancement method that explicitly incorporates domain-specific lexical knowledge into the BERT embedding space for the first time. Our approach introduces a supervised contrastive learning framework with a learnable projection head, designed to maximize intra-class similarity and inter-class dissimilarity. It jointly leverages an automatically constructed domain lexicon and context-aware word vector optimization. Extensive experiments on sentiment analysis, emotion recognition, and question answering classification demonstrate consistent and significant improvements over vanilla BERT and state-of-the-art fine-tuning baselines, achieving an average +3.2% F1-score gain under few-shot settings. The code and datasets are publicly released.

Technology Category

• Natural Language Processing: Lexical Semantics and Morphology
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Data Mining & Knowledge Management: Knowledge Acquisition from the Web

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Pre-trained language models such as BERT have been proved to be powerful in many natural language processing tasks. But in some text classification applications such as emotion recognition and sentiment analysis, BERT may not lead to satisfactory performance. This often happens in applications where keywords play critical roles in the prediction of class labels. Our investigation found that the root cause of the problem is that the context-based BERT embedding of the keywords may not be discriminative enough to produce discriminative text representation for classification. Motivated by this finding, we develop a method to enhance word embeddings using domain-specific lexical knowledge. The knowledge-based embedding enhancement model projects the BERT embedding into a new space where within-class similarity and between-class difference are maximized. To implement the knowledge-based word embedding enhancement model, we also develop a knowledge acquisition algorithm for automatically collecting lexical knowledge from online open sources. Experiment results on three classification tasks, including sentiment analysis, emotion recognition and question answering, have shown the effectiveness of our proposed word embedding enhancing model. The codes and datasets are in https://github.com/MidiyaZhu/KVWEFFER.

Problem

• Enhancing BERT embeddings for better text classification performance
• Improving discriminative power of keyword embeddings using domain knowledge
• Automating lexical knowledge acquisition from online sources for embedding enhancement

Innovation

• Enhances BERT embeddings using domain-specific lexical knowledge
• Projects embeddings to maximize within-class similarity and between-class difference
• Automatically acquires lexical knowledge from online open sources