Probing BERT for German Compound Semantics

2025-05-20 · Filip Miletić, Aaron Schmid, Sabine Schulte im Walde

This paper investigates the extent to which pretrained German BERT encodes knowledge of noun compound semantics. We comprehensively vary combinations of target tokens, layers, and cased vs. uncased models, and evaluate them by predicting the compositionality of 868 gold standard compounds. Looking at representational patterns within the transformer architecture, we observe trends comparable to equivalent prior work on English, with compositionality information most easily recoverable in the early layers. However, our strongest results clearly lag behind those reported for English, suggesting an inherently more difficult task in German. This may be due to the higher productivity of compounding in German than in English and the associated increase in constituent-level ambiguity, including in our target compound set.

📄 PDF Abstract BibTeX arXiv:2505.14130

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

Linear Warmup With Linear Decay Linear Warmup With Linear Decay is a learning rate schedule in which we increase the learning rate linearly for $n$ updates and then linearly decay afterwards.

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Attention Dropout Attention Dropout is a type of dropout used in attention-based architectures, where elements are randomly dropped out of the…

WordPiece 설명 없음

Refunds@Expedia|||How do I get a full refund from Expedia? “How do I get a full refund from Expedia? How do I get a full refund from Expedia? – Call ☎️ +1-(888) 829 (0881) or +1-805-330-4056 or +1-805-330-4056 for Quick Help &…

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Residual Connection 설명 없음

A Psycholinguistic Analysis of BERT's Representations of Compounds

2023-02-14 · Lars Buijtelaar, Sandro Pezzelle

This work studies the semantic representations learned by BERT for compounds, that is, expressions such as sunlight or bodyguard. We build on recent studies that explore semantic information in Transformers at the word l…

Probing for idiomaticity in vector space models

2021-04-01 · EACL 2021 2 · Marcos Garcia, Tiago Kramer Vieira, Carolina Scarton, Marco Idiart 외

Contextualised word representation models have been successfully used for capturing different word usages and they may be an attractive alternative for representing idiomaticity in language. In this paper, we propose pro…

What do we need to know about an unknown word when parsing German

2017-09-01 · WS 2017 9 · Bich-Ngoc Do, Ines Rehbein, Anette Frank

We propose a new type of subword embedding designed to provide more information about unknown compounds, a major source for OOV words in German. We present an extrinsic evaluation where we use the compound embeddings as …

Language Modelling POS Word Embeddings

Splitting Compounds by Semantic Analogy

2015-09-15 · WS 2015 1 · Joachim Daiber, Lautaro Quiroz, Roger Wechsler, Stella Frank

Compounding is a highly productive word-formation process in some languages that is often problematic for natural language processing applications. In this paper, we investigate whether distributional semantics in the fo…

Machine Translation Translation Word Embeddings

Chasing the Perfect Splitter: A Comparison of Different Compound Splitting Tools

2014-05-01 · LREC 2014 5 · Carla Parra Escart{\'\i}n

This paper reports on the evaluation of two compound splitters for German. Compounding is a very frequent phenomenon in German and thus efficient ways of detecting and correctly splitting compound words are needed for na…

Machine Translation Translation