Intrinsic Probing through Dimension Selection

2020-10-06 · EMNLP 2020 11 · Lucas Torroba Hennigen, Adina Williams, Ryan Cotterell

Most modern NLP systems make use of pre-trained contextual representations that attain astonishingly high performance on a variety of tasks. Such high performance should not be possible unless some form of linguistic structure inheres in these representations, and a wealth of research has sprung up on probing for it. In this paper, we draw a distinction between intrinsic probing, which examines how linguistic information is structured within a representation, and the extrinsic probing popular in prior work, which only argues for the presence of such information by showing that it can be successfully extracted. To enable intrinsic probing, we propose a novel framework based on a decomposable multivariate Gaussian probe that allows us to determine whether the linguistic information in word embeddings is dispersed or focal. We then probe fastText and BERT for various morphosyntactic attributes across 36 languages. We find that most attributes are reliably encoded by only a few neurons, with fastText concentrating its linguistic structure more than BERT.

📄 PDF Abstract BibTeX arXiv:2010.02812

Code (1)

rycolab/intrinsic-probing 공식 구현 pytorch

Tasks

Word Embeddings

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

fastText fastText embeddings exploit subword information to construct word embeddings. Representations are learnt of character $n$-grams, and words represented as the sum of the…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

WordPiece 설명 없음

Multi-Head Attention 설명 없음

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Linear Warmup With Linear Decay Linear Warmup With Linear Decay is a learning rate schedule in which we increase the learning rate linearly for $n$ updates and then linearly decay afterwards.

IdEst: Assessing Self-Supervised Learning Representations via Intrinsic Dimension

2026-06-02 · Julie Mordacq, Vicky Kalogeiton, Steve Oudot arxiv

Self-supervised learning (SSL) has emerged as a powerful paradigm for learning meaningful representations from unlabeled data. However, the standard protocol for evaluating these representations, linear probing, is compu…

Self-Supervised Learning

Minimax Lower Bound for Estimating Diffusion-based Local Intrinsic Dimension

2026-09-04 · Jaehee Seo, Wontae Jeong, Jisu Kim arxiv

While diffusion-based methods have recently emerged as effective tools for probing the intrinsic geometry of high-dimensional data, their statistical difficulty remains largely unexplored. We study estimation of the fini…

Sparse Geometric Representation Through Local Shape Probing

2018-07-01 · IEEE TVCG 2018 7 · Julie Digne, Sébastien Valette, Raphaëlle Chaine

We propose a new shape analysis approach based on the non-local analysis of local shape variations. Our method relies on a novel description of shape variations, called Local Probing Field (LPF), which describes how a lo…

Denoising Position

Probing with Noise: Unpicking the Warp and Weft of Embeddings

2022-10-21 · Filip Klubička, John D. Kelleher

Improving our understanding of how information is encoded in vector space can yield valuable interpretability insights. Alongside vector dimensions, we argue that it is possible for the vector norm to also carry linguist…

Sentence

Probing for sparse and fast variable selection with model-based boosting

2017-02-15 · Janek Thomas, Tobias Hepp, Andreas Mayr, Bernd Bischl

We present a new variable selection method based on model-based gradient boosting and randomly permuted variables. Model-based boosting is a tool to fit a statistical model while performing variable selection at the same…

Variable Selection