Does injecting linguistic structure into language models lead to better alignment with brain recordings?
Neuroscientists evaluate deep neural networks for natural language processing as possible candidate models for how language is processed in the brain. These models are often trained without explicit linguistic supervision, but have been shown to learn some linguistic structure in the absence of such supervision (Manning et al., 2020), potentially questioning the relevance of symbolic linguistic theories in modeling such cognitive processes (Warstadt and Bowman, 2020). We evaluate across two fMRI datasets whether language models align better with brain recordings, if their attention is biased by annotations from syntactic or semantic formalisms. Using structure from dependency or minimal recursion semantic annotations, we find alignments improve significantly for one of the datasets. For another dataset, we see more mixed results. We present an extensive analysis of these results. Our proposed approach enables the evaluation of more targeted hypotheses about the composition of meaning in the brain, expanding the range of possible scientific inferences a neuroscientist could make, and opens up new opportunities for cross-pollination between computational neuroscience and linguistics.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Mixture-of-Linguistic-Experts Adapters for Improving and Interpreting Pre-trained Language Models
In this work, we propose a method that combines two popular research areas by injecting linguistic structures into pre-trained language models in the parameter-efficient fine-tuning (PEFT) setting. In our approach, paral…
parameter-efficient fine-tuningTowards Decomposed Linguistic Representation with Holographic Reduced Representation
The vast majority of neural models in Natural Language Processing adopt a form of structureless distributed representations. While these models are powerful at making predictions, the representational form is rather crud…
FormOvercoming Barriers to Skill Injection in Language Modeling: Case Study in Arithmetic
Through their transfer learning abilities, highly-parameterized large pre-trained language models have dominated the NLP landscape for a multitude of downstream language tasks. Though linguistically proficient, the inabi…
Arithmetic ReasoningLanguage ModelingLanguage ModellingMathematical Reasoning+1Injecting linguistic knowledge into BERT for Dialogue State Tracking
Dialogue State Tracking (DST) models often employ intricate neural network architectures, necessitating substantial training data, and their inference process lacks transparency. This paper proposes a method that extract…
Decision MakingDialogue State TrackingEnhancing Linguistic Competence of Language Models through Pre-training with Language Learning Tasks
Language models (LMs) are pre-trained on raw text datasets to generate text sequences token-by-token. While this approach facilitates the learning of world knowledge and reasoning, it does not explicitly optimize for lin…
Language Acquisition