The Importance of Encoder Choice:A Tabular-Image Study

2026-07-08

AI Summary

Existing multimodal approaches for image-tabular data typically employ simplistic MLP-based tabular encoders, which limits overall performance. This work introduces state-of-the-art tabular models—including label-dependent contextual learning methods—into multimodal frameworks for the first time and proposes a unified embedding strategy that effectively mitigates inconsistencies arising from differing label availability during training and inference. Through systematic evaluation of various combinations of state-of-the-art tabular and image encoders, the study demonstrates that the choice of tabular encoder critically influences multimodal performance. These findings establish a new paradigm and provide empirical grounding for joint image-tabular modeling.

Abstract

Multimodal learning usually requires a dedicated encoder per modality. When a tabular modality is involved, prior work has been mostly using a \emph{plain MLP} as the encoder. Yet if it were a strong encoder, the tabular domain would not be ``the last unconquered castle for deep learning''. This study evaluates state-of-the-art tabular models as encoders in the image-tabular setting for the first time. An obstacle stands out. In-Context Learning models, among the best performing methods in the tabular domain, require labels to process instances, making it non-trivial to embed training and test instances the same way. We addressed this problem across multiple models of this family. With this study, we would like to highlight the importance of encoder factor in the multimodal learning.

Problem

• multimodal learning
• tabular data
• encoder choice
• In-Context Learning
• image-tabular

Innovation

• tabular encoder
• multimodal learning
• In-Context Learning
• image-tabular fusion
• deep learning for tabular data