Long Warm-up and Self-Training: Training Strategies of NICT-2 NMT System at WAT-2019

2019-11-01 · WS 2019 11 · Kenji Imamura, Eiichiro Sumita

This paper describes the NICT-2 neural machine translation system at the 6th Workshop on Asian Translation. This system employs the standard Transformer model but features the following two characteristics. One is the long warm-up strategy, which performs a longer warm-up of the learning rate at the start of the training than conventional approaches. Another is that the system introduces self-training approaches based on multiple back-translations generated by sampling. We participated in three tasks{---}ASPEC.en-ja, ASPEC.ja-en, and TDDC.ja-en{---}using this system.

📄 PDF Abstract BibTeX

Code (0)

등록된 구현이 없습니다.

Tasks

Machine TranslationNMTTranslation

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Position-Wise Feed-Forward Layer 설명 없음

Residual Connection 설명 없음

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

ReLU How Do I Communicate to Expedia? How Do I Communicate to Expedia? – Call ☎️ +1-(888) 829 (0881) or +1-805-330-4056 or +1-805-330-4056 for Live Support & Special Travel…

Learning Strategies in Particle Swarm Optimizer: A Critical Review and Performance Analysis

2025-04-16 · Dikshit Chauhan, Shivani, P. N. Suganthan

Nature has long inspired the development of swarm intelligence (SI), a key branch of artificial intelligence that models collective behaviors observed in biological systems for solving complex optimization problems. Part…

Continual Pre-Training of Large Language Models: How to (re)warm your model?

2023-08-08 · Kshitij Gupta, Benjamin Thérien, Adam Ibrahim, Mats L. Richter 외

Large language models (LLMs) are routinely pre-trained on billions of tokens, only to restart the process over again once new data becomes available. A much cheaper and more efficient solution would be to enable the cont…

Language Modelling

Self-Adaptive Swarm System (SASS)

2021-05-25 · Qin Yang

Distributed artificial intelligence (DAI) studies artificial intelligence entities working together to reason, plan, solve problems, organize behaviors and strategies, make collective decisions and learn. This Ph.D. rese…

Decision Making

When Will Gradient Regularization Be Harmful?

2024-06-14 · Yang Zhao, Hao Zhang, Xiuyuan Hu

Gradient regularization (GR), which aims to penalize the gradient norm atop the loss function, has shown promising results in training modern over-parameterized deep neural networks. However, can we trust this powerful t…

DiGA: Distil to Generalize and then Adapt for Domain Adaptive Semantic Segmentation

2023-04-05 · CVPR 2023 1 · Fengyi Shen, Akhil Gurram, Ziyuan Liu, He Wang 외

Domain adaptive semantic segmentation methods commonly utilize stage-wise training, consisting of a warm-up and a self-training stage. However, this popular approach still faces several challenges in each stage: for warm…

Data Augmentation Knowledge Distillation Pseudo Label Semantic Segmentation