When a sentence does not introduce a discourse entity, Transformer-based models still sometimes refer to it
Understanding longer narratives or participating in conversations requires tracking of discourse entities that have been mentioned. Indefinite noun phrases (NPs), such as 'a dog', frequently introduce discourse entities but this behavior is modulated by sentential operators such as negation. For example, 'a dog' in 'Arthur doesn't own a dog' does not introduce a discourse entity due to the presence of negation. In this work, we adapt the psycholinguistic assessment of language models paradigm to higher-level linguistic phenomena and introduce an English evaluation suite that targets the knowledge of the interactions between sentential operators and indefinite NPs. We use this evaluation suite for a fine-grained investigation of the entity tracking abilities of the Transformer-based models GPT-2 and GPT-3. We find that while the models are to a certain extent sensitive to the interactions we investigate, they are all challenged by the presence of multiple NPs and their behavior is not systematic, which suggests that even models at the scale of GPT-3 do not fully acquire basic entity tracking abilities.
Code (1)
Tasks
NegationSentenceMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
When a sentence does not introduce a discourse entity, Transformer-based models still often refer to it
Understanding longer narratives or participating in conversations requires tracking of discourse entities that have been mentioned. Indefinite noun phrases, such as 'a dog', frequently introduce discourse entities but th…
NegationSentenceEntity-Augmented Distributional Semantics for Discourse Relations
Discourse relations bind smaller linguistic elements into coherent texts. However, automatically identifying discourse relations is difficult, because it requires understanding the semantics of the linked sentences. A mo…
RelationSentenceAn entity-driven recursive neural network model for chinese discourse coherence modeling
Chinese discourse coherence modeling remains a challenge taskin Natural Language Processing field.Existing approaches mostlyfocus on the need for feature engineering, whichadoptthe sophisticated features to capture the l…
Coherence EvaluationFeature EngineeringMachine TranslationSentence+2Narrative-UFET: Narrative Generation for Ultra-Fine Entity Typing
Ultra-fine entity typing (UFET) assigns highly specific types to entity mentions, but current approaches struggle with types in the long tail. We hypothesize that a key limitation is the reliance on sentence-level contex…
Entity TypingNew or Old? Exploring How Pre-Trained Language Models Represent Discourse Entities
Recent research shows that pre-trained language models, built to generate text conditioned on some context, learn to encode syntactic knowledge to a certain degree. This has motivated researchers to move beyond the sente…
Binary ClassificationSentence