Adaptable Multi-Domain Language Model for Transformer ASR
We propose an adapter based multi-domain Transformer based language model (LM) for Transformer ASR. The model consists of a big size common LM and small size adapters. The model can perform multi-domain adaptation with only the small size adapters and its related layers. The proposed model can reuse the full fine-tuned LM which is fine-tuned using all layers of an original model. The proposed LM can be expanded to new domains by adding about 2% of parameters for a first domain and 13% parameters for after second domain. The proposed model is also effective in reducing the model maintenance cost because it is possible to omit the costly and time-consuming common LM pre-training process. Using proposed adapter based approach, we observed that a general LM with adapter can outperform a dedicated music domain LM in terms of word error rate (WER).
Code (0)
등록된 구현이 없습니다.
Tasks
Domain AdaptationLanguage ModelingLanguage ModellingmodelMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Hierarchical Autoregressive Transformers: Combining Byte- and Word-Level Processing for Robust, Adaptable Language Models
Tokenization is a fundamental step in natural language processing, breaking text into units that computational models can process. While learned subword tokenizers have become the de-facto standard, they present challeng…
Language ModellingWord EmbeddingsModRWKV: Transformer Multimodality in Linear Time
Currently, most multimodal studies are based on large language models (LLMs) with quadratic-complexity Transformer architectures. While linear models like RNNs enjoy low inference costs, their application has been largel…
Computational EfficiencyAdaptable and Reliable Text Classification using Large Language Models
Text classification is fundamental in Natural Language Processing (NLP), and the advent of Large Language Models (LLMs) has revolutionized the field. This paper introduces an adaptable and reliable text classification pa…
ClassificationMulti-Label ClassificationMUlTI-LABEL-ClASSIFICATIONSentiment Analysis+2PARAGRAPH2GRAPH: A GNN-based framework for layout paragraph analysis
Document layout analysis has a wide range of requirements across various domains, languages, and business scenarios. However, most current state-of-the-art algorithms are language-dependent, with architectures that rely …
Document Layout AnalysisGraph Neural NetworkImage SegmentationSemantic SegmentationGuiding Symbolic Natural Language Grammar Induction via Transformer-Based Sequence Probabilities
A novel approach to automated learning of syntactic rules governing natural languages is proposed, based on using probabilities assigned to sentences (and potentially longer word sequences) by transformer neural network …
Clustering