TypeProbe: Recovering Type Representations from Hidden States of Pre-trained Code Models

2026-07-09

AI Summary

This study investigates whether pretrained code models encode cross-lingually consistent formal type semantics in their hidden representations. To this end, the authors construct a parallel Java–Python code dataset and employ linear probing on residual stream activations to analyze how type information is represented. They further design cross-lingual transfer experiments to assess whether models can recover formal type annotations from untyped code. This work presents the first direct interpretability analysis targeting formal type semantics and cross-lingual representation alignment in pretrained models. The results demonstrate that such models indeed learn transferable, cross-lingually aligned type structures, and that these representations exhibit robustness to lexical perturbations and syntactic discrepancies between languages.

Abstract

State-of-the-art code models achieve impressive performance, yet the extent to which they internally encode type information remains poorly understood. We probe the residual streams of pretrained code models for internal type representations using a parallel dataset of Java and Python code examples. Our results show that cross-lingual type representations emerge even from untyped code. Moreover, we test whether hidden states linearly encode the result type implied by typed function application by training probes on one language to infer argument and result types in the other. Finally, we find that this structure is partly robust to lexical perturbations and cross-language syntactic variations. To the best of our knowledge, prior work on interpretability of code models has not directly targeted formal type semantics or cross-lingual type representations. We release our code and datasets.

Problem

• type representations
• pre-trained code models
• cross-lingual
• formal type semantics
• interpretability

Innovation

• type probing
• pre-trained code models
• cross-lingual representation
• type semantics
• interpretability