Attention Quantization for Tabular Foundation Models

2026-09-11

AI Summary

研究针对表格基础模型的推理性能优化问题，提出一种将注意力机制中的查询、键和值量化为FP8的方法，并使用FP8矩阵乘法指令加速计算。

Abstract

With the recent rise and adoption of tabular foundation models, optimizing their inference performance becomes an emerging field for efficiency research. While the models are architecturally similar to transformer-based large language models (LLMs), the size and serving patterns differ significantly. We show that the focus should be on the attention calculation and less on weight or KV cache quantization, which are more popular in LLMs. We develop a quantization strategy for queries, keys, and values to FP8 and use explicit FP8 matrix multiplication instructions to speed up the attention calculation. We find that it is crucial to align the quantization error in the test rows with the quantization error in the training rows, as otherwise the accuracy drops drastically. Our Triton kernel achieves a speedup up to 1.7x over regular 16-bit kernels, and we show that on TabPFN-v3 and TabICLv2 there is no relevant accuracy loss across TabArena and BeyondArena.

Problem

• attention quantization
• tabular foundation models
• inference performance
• FP8
• accuracy

Innovation

• Attention Quantization
• FP8
• Matrix Multiplication Instructions
• Quantization Error Alignment