Forecasting Credit Ratings: A Case Study where Traditional Methods Outperform Generative LLMs

2024-07-24

AI Summary

This study addresses corporate credit rating prediction by systematically comparing generative large language models (LLMs)—including Llama-2 and GPT-3.5—against traditional machine learning (XGBoost) in modeling heterogeneous multimodal data (textual, financial, and macroeconomic). Experiments reveal that current LLMs underperform XGBoost significantly on numerically intensive financial forecasting, challenging their presumed universality for structured financial tasks. To bridge this gap, we propose a novel collaborative modeling paradigm: deeply fusing high-fidelity textual embeddings (generated via Sentence-BERT) with structured financial and macroeconomic features, then feeding the joint representation into XGBoost for end-to-end learning. Our approach consistently outperforms both zero-shot and fine-tuned LLM baselines in rating accuracy and F1-score—especially in cross-grade transition scenarios (e.g., BBB→BB)—with improvements up to 12.6%. It further ensures strong model interpretability and deployment robustness.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large Language Models (LLMs) have been shown to perform well for many downstream tasks. Transfer learning can enable LLMs to acquire skills that were not targeted during pre-training. In financial contexts, LLMs can sometimes beat well-established benchmarks. This paper investigates how well LLMs perform in the task of forecasting corporate credit ratings. We show that while LLMs are very good at encoding textual information, traditional methods are still very competitive when it comes to encoding numeric and multimodal data. For our task, current LLMs perform worse than a more traditional XGBoost architecture that combines fundamental and macroeconomic data with high-density text-based embedding features.

Problem

• Large Language Models
• Credit Rating Prediction
• Comparative Accuracy

Innovation

• XGBoost
• Diverse Data Integration
• Corporate Credit Ratings Prediction