Attention Understands Semantic Relations
Today, natural language processing heavily relies on pre-trained large language models. Even though such models are criticized for the poor interpretability, they still yield state-of-the-art solutions for a wide set of very different tasks. While lots of probing studies have been conducted to measure the models’ awareness of grammatical knowledge, semantic probing is less popular. In this work, we introduce the probing pipeline to study the representedness of semantic relations in transformer language models. We show that in this task, attention scores are nearly as expressive as the layers’ output activations, despite their lesser ability to represent surface cues. This supports the hypothesis that attention mechanisms are focusing not only on the syntactic relational information but also on the semantic one.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Construction and Evaluation of a Self-Attention Model for Semantic Understanding of Sentence-Final Particles
Sentence-final particles serve an essential role in spoken Japanese because they express the speaker's mental attitudes toward a proposition and/or an interlocutor. They are acquired at early ages and occur very frequent…
SentenceRelation/Entity-Centric Reading Comprehension
Constructing a machine that understands human language is one of the most elusive and long-standing challenges in artificial intelligence. This thesis addresses this challenge through studies of reading comprehension wit…
Question AnsweringReading ComprehensionRelationExploiting Emotion-Semantic Correlations for Empathetic Response Generation
Empathetic response generation aims to generate empathetic responses by understanding the speaker's emotional feelings from the language of dialogue. Recent methods capture emotional words in the language of communicator…
Dialogue GenerationEmpathetic Response GenerationResponse GenerationSemantic Diversity-aware Prototype-based Learning for Unbiased Scene Graph Generation
The scene graph generation (SGG) task involves detecting objects within an image and predicting predicates that represent the relationships between the objects. However, in SGG benchmark datasets, each subject-object pai…
DiversityGraph GenerationScene Graph GenerationUnbiased Scene Graph GenerationLean Finder: Semantic Search for Mathlib That Understands User Intents
We present Lean Finder, a semantic search engine for Lean and mathlib that understands and aligns with the intents of mathematicians. Progress in formal theorem proving is often hindered by the difficulty of locating rel…