Multi-objective Representation for Numbers in Clinical Narratives: A CamemBERT-Bio-Based Alternative to Large-Scale LLMs

2024-05-28

AI Summary

This work addresses the challenge of jointly modeling numerical semantics and magnitude in medical texts using Transformer-based models. The task involves classifying clinical numeric values into eight fine-grained physiological categories. To this end, we propose a lightweight and efficient approach centered on two novel components: (1) Label-Embedded Self-Attention (LESA), which injects category priors directly into self-attention computation; and (2) Xval, a numerical representation enhancement module that enables context-aware and magnitude-sensitive multi-objective encoding. Our method requires no large language model and is trainable on small-scale, domain-specific medical data. Experiments demonstrate that LESA alone improves F1-score by over 13%; when combined with Xval, the full model achieves performance comparable to GPT-4—significantly outperforming standard fine-tuning and conventional approaches. This establishes a new paradigm for medical numerical understanding in resource-constrained settings.

Technology Category

• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Calibration & Uncertainty Quantification
• Computer Vision: Multi-modal Vision

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search

Abstract

The processing of numerical values is a rapidly developing area in the field of Language Models (LLMs). Despite numerous advancements achieved by previous research, significant challenges persist, particularly within the healthcare domain. This paper investigates the limitations of Transformer models in understanding numerical values. extit{Objective:} this research aims to categorize numerical values extracted from medical documents into eight specific physiological categories using CamemBERT-bio. extit{Methods:} In a context where scalable methods and Large Language Models (LLMs) are emphasized, we explore lifting the limitations of transformer-based models. We examine two strategies: fine-tuning CamemBERT-bio on a small medical dataset, integrating Label Embedding for Self-Attention (LESA), and combining LESA with additional enhancement techniques such as Xval. Given that CamemBERT-bio is already pre-trained on a large medical dataset, the first approach aims to update its encoder with the newly added label embeddings technique. In contrast, the second approach seeks to develop multiple representations of numbers (contextual and magnitude-based) to achieve more robust number embeddings. extit{Results:} As anticipated, fine-tuning the standard CamemBERT-bio on our small medical dataset did not improve F1 scores. However, significant improvements were observed with CamemBERT-bio + LESA, resulting in an over 13% increase. Similar enhancements were noted when combining LESA with Xval, outperforming conventional methods and giving comparable results to GPT-4 extit{Conclusions and Novelty:} This study introduces two innovative techniques for handling numerical data, which are also applicable to other modalities. We illustrate how these techniques can improve the performance of Transformer-based models, achieving more reliable classification results even with small datasets.

Problem

• Limitations of Transformer models in understanding numerical values.
• Categorizing numerical values from medical documents into physiological categories.
• Improving Transformer-based models' performance with small medical datasets.

Innovation

• Fine-tunes CamemBERT-bio with LESA for numerical data.
• Combines LESA with Xval for enhanced number embeddings.
• Achieves robust classification with small medical datasets.