Syntactic Inductive Bias in Transformer Language Models: Especially Helpful for Low-Resource Languages?
A line of work on Transformer-based language models such as BERT has attempted to use syntactic inductive bias to enhance the pretraining process, on the theory that building syntactic structure into the training process should reduce the amount of data needed for training. But such methods are often tested for high-resource languages such as English. In this work, we investigate whether these methods can compensate for data sparseness in low-resource languages, hypothesizing that they ought to be more effective for low-resource languages. We experiment with five low-resource languages: Uyghur, Wolof, Maltese, Coptic, and Ancient Greek. We find that these syntactic inductive bias methods produce uneven results in low-resource settings, and provide surprisingly little benefit in most cases.
Code (1)
Tasks
Inductive BiasMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Strengthening Structural Inductive Biases by Pre-training to Perform Syntactic Transformations
Models need appropriate inductive biases to effectively learn from small amounts of data and generalize systematically outside of the training distribution. While Transformers are highly versatile and powerful, they can …
ChunkingFew-Shot LearningInductive BiasSemantic ParsingSneaking Syntax into Transformer Language Models with Tree Regularization
While compositional accounts of human language understanding are based on a hierarchical tree-like process, neural models like transformers lack a direct inductive bias for such tree structures. Introducing syntactic ind…
Inductive BiasHow to Plant Trees in Language Models: Data and Architectural Effects on the Emergence of Syntactic Inductive Biases
Accurate syntactic representations are essential for robust generalization in natural language. Recent work has found that pre-training can teach language models to rely on hierarchical syntactic features - as opposed to…
DecoderInductive BiasLanguage AcquisitionSyntactic Inductive Biases for Deep Learning Methods
In this thesis, we try to build a connection between the two schools by introducing syntactic inductive biases for deep learning models. We propose two families of inductive biases, one for constituency structure and ano…
Deep LearningInductive BiasControlled Evaluation of Grammatical Knowledge in Mandarin Chinese Language Models
Prior work has shown that structural supervision helps English language models learn generalizations about syntactic phenomena such as subject-verb agreement. However, it remains unclear if such an inductive bias would a…
Inductive Bias