Code and Pixels: Multi-Modal Contrastive Pre-training for Enhanced Tabular Data Analysis

2025-01-13

AI Summary

To address the weak representation capability of pure tabular data and the limited modeling performance in the absence of image modality, this paper proposes the Multi-Task Contrastive Masked Tabular Modeling (MT-CMTM) framework. MT-CMTM introduces a novel dual-path pretraining paradigm that jointly integrates cross-modal contrastive learning and masked column reconstruction. It employs a lightweight 1D-ResNet-CBAM architecture to enable image-free, tabular-specific representation learning. By jointly optimizing table–image cross-modal alignment and masked column reconstruction, the method significantly enhances structured semantic capture. Extensive experiments on the HIPMP and DVM benchmarks demonstrate that MT-CMTM achieves a 1.48% relative reduction in MSE and a 2.38% absolute improvement in accuracy over prior methods—marking the first empirical validation of cross-modal pretraining’s effectiveness for downstream tasks on pure tabular data.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

Learning from tabular data is of paramount importance, as it complements the conventional analysis of image and video data by providing a rich source of structured information that is often critical for comprehensive understanding and decision-making processes. We present Multi-task Contrastive Masked Tabular Modeling (MT-CMTM), a novel method aiming to enhance tabular models by leveraging the correlation between tabular data and corresponding images. MT-CMTM employs a dual strategy combining contrastive learning with masked tabular modeling, optimizing the synergy between these data modalities. Central to our approach is a 1D Convolutional Neural Network with residual connections and an attention mechanism (1D-ResNet-CBAM), designed to efficiently process tabular data without relying on images. This enables MT-CMTM to handle purely tabular data for downstream tasks, eliminating the need for potentially costly image acquisition and processing. We evaluated MT-CMTM on the DVM car dataset, which is uniquely suited for this particular scenario, and the newly developed HIPMP dataset, which connects membrane fabrication parameters with image data. Our MT-CMTM model outperforms the proposed tabular 1D-ResNet-CBAM, which is trained from scratch, achieving a relative 1.48% improvement in relative MSE on HIPMP and a 2.38% increase in absolute accuracy on DVM. These results demonstrate MT-CMTM's robustness and its potential to advance the field of multi-modal learning.

Problem

• Structured Data Learning
• Computer Decision-making
• Tabular Data Analysis

Innovation

• MT-CMTM
• 1D-ResNet-CBAM
• Multi-modal Learning