Retrofitting Structure-aware Transformer Language Model for End Tasks
We consider retrofitting structure-aware Transformer-based language model for facilitating end tasks by proposing to exploit syntactic distance to encode both the phrasal constituency and dependency connection into the language model. A middle-layer structural learning strategy is leveraged for structure integration, accomplished with main semantic task training under multi-task learning scheme. Experimental results show that the retrofitted structure-aware Transformer language model achieves improved perplexity, meanwhile inducing accurate syntactic phrases. By performing structure-aware fine-tuning, our model achieves significant improvements for both semantic- and syntactic-dependent tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingMulti-Task LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Capturing Regional Variation with Distributed Place Representations and Geographic Retrofitting
Dialects are one of the main drivers of language variation, a major challenge for natural language processing tools. In most languages, dialects exist along a continuum, and are commonly discretized by combining the exte…
ClusteringDimensionality ReductionMachine TranslationRepresentation LearningRetrofitting Light-weight Language Models for Emotions using Supervised Contrastive Learning
We present a novel retrofitting method to induce emotion aspects into pre-trained language models (PLMs) such as BERT and RoBERTa. Our method updates pre-trained network weights using contrastive learning so that the tex…
Contrastive LearningFew-Shot LearningRetrievalSarcasm Detection+1Expansional Retrofitting for Word Vector Enrichment
Retrofitting techniques, which inject external resources into word representations, have compensated the weakness of distributed representations in semantic and relational knowledge between words. Implicitly retrofitting…
General Classificationtext-classificationText ClassificationWord SimilarityDifferentiable Hierarchical Visual Tokenization
Vision Transformers rely on fixed patch tokens that ignore the spatial and semantic structure of images. In this work, we introduce an end-to-end differentiable tokenizer that adapts to image content with pixel-level gra…
Retrofitting Distributional Embeddings to Knowledge Graphs with Functional Relations
Knowledge graphs are a versatile framework to encode richly structured data relationships, but it can be challenging to combine these graphs with unstructured data. Methods for retrofitting pre-trained entity representat…
Knowledge Graph CompletionKnowledge Graphs