Probing for Referential Information in Language Models
Language models keep track of complex information about the preceding context {--} including, e.g., syntactic relations in a sentence. We investigate whether they also capture information beneficial for resolving pronominal anaphora in English. We analyze two state of the art models with LSTM and Transformer architectures, via probe tasks and analysis on a coreference annotated corpus. The Transformer outperforms the LSTM in all analyses. Our results suggest that language models are more successful at learning grammatical constraints than they are at learning truly referential information, in the sense of capturing the fact that we use language to refer to entities in the world. However, we find traces of the latter aspect, too.
Code (0)
등록된 구현이 없습니다.
Tasks
SentenceMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
What can Neural Referential Form Selectors Learn?
Despite achieving encouraging results, neural Referring Expression Generation models are often thought to lack transparency. We probed neural Referential Form Selection (RFS) models to find out to what extent the linguis…
FormPositionReferring ExpressionReferring expression generation+1Assessing Neural Referential Form Selectors on a Realistic Multilingual Dataset
Previous work on Neural Referring Expression Generation (REG) all uses WebNLG, an English dataset that has been shown to reflect a very limited range of referring expression (RE) use. To tackle this issue, we build a dat…
FormReferring ExpressionReferring expression generationEmCom-Diffusion: Probing Visual Reflection in Emergent Languages via Image Generation
Measuring the extent to which emergent languages encode the visual content of their inputs is an open problem. We refer to this property as visual reflection: the extent to which emergent messages preserve information ab…
Image GenerationVisuospatial Perspective Taking in Multimodal Language Models
As multimodal language models (MLMs) are increasingly used in social and collaborative settings, it is crucial to evaluate their perspective-taking abilities. Existing benchmarks largely rely on text-based vignettes or s…
Scene UnderstandingDistinct dynamics of conceptual and referential disruptions in human reading and large language model processing
Linguistic meaning is grounded in conceptual content, from which reference to particular entities emerges as words enter discourse. To examine the processing dynamics associated with these two dimensions of meaning, we s…