How Should Transformers Encode Numeric Values in Electronic Health Records?

2026-07-01

AI Summary

This study addresses the challenge of effectively encoding numerical data in Transformer-based models for electronic health record (EHR) sequence modeling by systematically comparing discrete, continuous, and hybrid encoding strategies. The authors propose a hybrid tokenization approach as a practical default and evaluate it end-to-end on both synthetic arithmetic tasks and real-world clinical prediction benchmarks. Key findings include an empirical power-law relationship between optimal binning resolution and dataset size, evidence that models favor “good enough” over exact numerical computation, and consistent multi-task performance gains with the hybrid method. Notably, the utility of laboratory value encoding is highly task-dependent, collectively supporting a design principle that prioritizes practical effectiveness over maximal numerical precision.

Abstract

How do we encode numeric values in transformer-based sequence processing, particularly in electronic health record (EHR) data? We systematically compare discrete, continuous, and hybrid value encoding strategies using synthetic arithmetic tasks embedded within real-world EHR data, as well as real-world clinical prediction tasks. Our study reveals trade-offs between numeric precision, optimisation stability, and architectural flexibility. We find that approaches that explicitly model value-concept interactions perform best on precision-sensitive arithmetic tasks when architectural constraints permit. Hybrid token-based approaches that retain numeric values but apply binning prior to projection provide a more robust and broadly applicable alternative, with the optimal number of bins following a simple empirically derived power-law in dataset size. Across tasks, models consistently exhibit reliable "good enough" numeric computation rather than exact arithmetic, while clinical gains from incorporating laboratory values are task-dependent. This suggests that robustness and deployability often outweigh maximal numeric precision in practice, motivating hybrid token-based approaches as a practical default.

Problem

• numeric encoding
• transformers
• electronic health records
• value representation
• sequence modeling

Innovation

• numeric encoding
• transformer architecture
• electronic health records
• hybrid tokenization
• value-concept interaction