A Structural Probe for Finding Syntax in Word Representations
Recent work has improved our ability to detect linguistic knowledge in word representations. However, current methods for detecting syntactic knowledge do not test whether syntax trees are represented in their entirety. In this work, we propose a structural probe, which evaluates whether syntax trees are embedded in a linear transformation of a neural network{'}s word representation space. The probe identifies a linear transformation under which squared L2 distance encodes the distance between words in the parse tree, and one in which squared L2 norm encodes depth in the parse tree. Using our probe, we show that such transformations exist for both ELMo and BERT but not in baselines, providing evidence that entire syntax trees are embedded implicitly in deep models{'} vector geometry.
Code (1)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Probing Syntax in Large Language Models: Successes and Remaining Challenges
The syntactic structures of sentences can be readily read-out from the activations of large language models (LLMs). However, the ``structural probes'' that have been developed to reveal this phenomenon are typically eval…
A polar coordinate system represents syntax in large language models
Originally formalized with symbolic representations, syntactic trees may also be effectively represented in the activations of large language models (LLMs). Indeed, a 'Structural Probe' can find a subspace of neural acti…
Word EmbeddingsProbing BERT in Hyperbolic Spaces
Recently, a variety of probing tasks are proposed to discover linguistic properties learned in contextualized word embeddings. Many of these works implicitly assume these embeddings lay in certain metric spaces, typicall…
Word EmbeddingsWhen Does Syntax Mediate Neural Language Model Performance? Evidence from Dropout Probes
Recent causal probing literature reveals when language models and syntactic probes use similar representations. Such techniques may yield "false negative" causality results: models may use representations of syntax, but …
Language ModelingLanguage ModellingWhen Does Syntax Mediate Neural Language Model Performance? Evidence from Dropout Probes
Recent causal probing literature reveals when language models and syntactic probes use similar representations. Such techniques may yield ``false negative'' causality results: models may use representations of syntax, bu…
Language ModelingLanguage Modelling