Optimizing Deep Transformers for Chinese-Thai Low-Resource Translation

2022-12-24 · Wenjie Hao, Hongfei Xu, Lingling Mu, Hongying Zan

In this paper, we study the use of deep Transformer translation model for the CCMT 2022 Chinese-Thai low-resource machine translation task. We first explore the experiment settings (including the number of BPE merge operations, dropout probability, embedding size, etc.) for the low-resource scenario with the 6-layer Transformer. Considering that increasing the number of layers also increases the regularization on new model parameters (dropout modules are also introduced when using more layers), we adopt the highest performance setting but increase the depth of the Transformer to 24 layers to obtain improved translation quality. Our work obtains the SOTA performance in the Chinese-to-Thai translation in the constrained evaluation.

📄 PDF Abstract BibTeX arXiv:2212.12662

Code (0)

등록된 구현이 없습니다.

Tasks

Machine Translation Translation

Methods 이 논문이 사용한 방법론

Multi-Head Attention 설명 없음

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Adam 설명 없음

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Residual Connection 설명 없음

Xmodel-1.5: An 1B-scale Multilingual LLM

2024-11-15 · Wang Qun, Liu Yang, Lin Qingquan, Jiang Ling

We introduce Xmodel-1.5, a 1-billion-parameter multilingual large language model pretrained on 2 trillion tokens, designed for balanced performance and scalability. Unlike most large models that use the BPE tokenizer, Xm…

Language Modeling Language Modelling Large Language Model

An Overview of BPPT's Indonesian Language Resources

2016-12-01 · WS 2016 12 · Gunarso Gunarso, Hammam Riza

This paper describes various Indonesian language resources that Agency for the Assessment and Application of Technology (BPPT) has developed and collected since mid 80{'}s when we joined MMTS (Multilingual Machine Transl…

Machine Translation speech-recognition Speech Recognition Speech Synthesis+1

Multiple Segmentations of Thai Sentences for Neural Machine Translation

2020-04-23 · LREC 2020 5 · Alberto Poncelas, Wichaya Pidchamook, Chao-Hong Liu, James Hadley 외

Thai is a low-resource language, so it is often the case that data is not available in sufficient quantities to train an Neural Machine Translation (NMT) model which perform to a high level of quality. In addition, the T…

Machine Translation NMT Sentence Translation

Can General-Purpose Large Language Models Generalize to English-Thai Machine Translation ?

2024-10-22 · Jirat Chiaranaipanich, Naiyarat Hanmatheekuna, Jitkapat Sawatphol, Krittamate Tiankanon 외

Large language models (LLMs) perform well on common tasks but struggle with generalization in low-resource and low-computation settings. We examine this limitation by testing various LLMs and specialized translation mode…

Machine Translation Quantization Translation

Typhoon: Thai Large Language Models

2023-12-21 · Kunat Pipatanakul, Phatrasek Jirabovonvisut, Potsawee Manakul, Sittipong Sripaisarnmongkol 외

Typhoon is a series of Thai large language models (LLMs) developed specifically for the Thai language. This technical report presents challenges and insights in developing Thai LLMs, including data preparation, pretraini…

Question Answering World Knowledge