Combining Deep Generative Models and Multi-lingual Pretraining for Semi-supervised Document Classification
Semi-supervised learning through deep generative models and multi-lingual pretraining techniques have orchestrated tremendous success across different areas of NLP. Nonetheless, their development has happened in isolation, while the combination of both could potentially be effective for tackling task-specific labelled data shortage. To bridge this gap, we combine semi-supervised deep generative models and multi-lingual pretraining to form a pipeline for document classification task. Compared to strong supervised learning baselines, our semi-supervised classification framework is highly competitive and outperforms the state-of-the-art counterparts in low-resource settings across several languages.
Code (1)
Tasks
ClassificationDocument ClassificationGeneral ClassificationSimilar Papers 제목 키워드 기반
FinGPT: Large Generative Models for a Small Language
Large language models (LLMs) excel in many tasks in NLP and beyond, but most open models have very limited coverage of smaller languages and LLM work tends to focus on languages where nearly unlimited data is available f…
DOCmT5: Document-Level Pretraining of Multilingual Language Models
In this paper, we introduce DOCmT5, a multilingual sequence-to-sequence language model pretrained with large scale parallel documents. While previous approaches have focused on leveraging sentence-level parallel data, we…
de-enDocument SummarizationDocument TranslationLanguage Modeling+4Revisiting Multilingual Data Mixtures in Language Model Pretraining
The impact of different multilingual data mixtures in pretraining large language models (LLMs) has been a topic of ongoing debate, often raising concerns about potential trade-offs between language coverage and model per…
LlamaTurk: Adapting Open-Source Generative Large Language Models for Low-Resource Language
Despite advancements in English-dominant generative large language models, further development is needed for low-resource languages to enhance global accessibility. The primary methods for representing these languages ar…
Cross-lingual Language Model Pretraining
Recent studies have demonstrated the efficiency of generative pretraining for English natural language understanding. In this work, we extend this approach to multiple languages and show the effectiveness of cross-lingua…
Language ModelingLanguage ModellingMachine Translationmodel+3