TypeProbe: Recovering Type Representations from Hidden States of Pre-trained Code Models
State-of-the-art code models achieve impressive performance, yet the extent to which they internally encode type information remains poorly understood. We probe the residual streams of pretrained code models for internal type representations using a parallel dataset of Java and Python code examples. Our results show that cross-lingual type representations emerge even from untyped code. Moreover, we test whether hidden states linearly encode the result type implied by typed function application by training probes on one language to infer argument and result types in the other. Finally, we find that this structure is partly robust to lexical perturbations and cross-language syntactic variations. To the best of our knowledge, prior work on interpretability of code models has not directly targeted formal type semantics or cross-lingual type representations. We release our code and datasets.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Vector Arithmetic in Concept and Token Subspaces
In order to predict the next token, LLMs must represent semantic and surface-level information about the current word. Previous work identified two types of attention heads that disentangle this information: (i) Concept …
Recovering Input Text from Hidden States: Study of Gradient-Based Inversion of Decoder-Only Language Models
This work studies the hidden-state inversion problem: recovering the original input token sequence of a decoder-only language model from its last-layer hidden states. Rather than treating inversion as a one-shot reconstr…
Evaluation of LLMs-based Hidden States as Author Representations for Psychological Human-Centered NLP Tasks
Like most of NLP, models for human-centered NLP tasks -- tasks attempting to assess author-level information -- predominantly use representations derived from hidden states of Transformer-based LLMs. However, what compon…
Recovering the state and dynamics of autonomous system with partial states solution using neural networks
In this paper we explore the performance of deep hidden physics model (M. Raissi 2018) for autonomous systems. These systems are described by set of ordinary differential equations which do not explicitly depend on time.…
AST-Probe: Recovering abstract syntax trees from hidden representations of pre-trained language models
The objective of pre-trained language models is to learn contextual representations of textual data. Pre-trained language models have become mainstream in natural language processing and code modeling. Using probes, a te…