Fast Training of NMT Model with Data Sorting

2023-08-16 · Daniela N. Rim, Kimera Richard, Heeyoul Choi

The Transformer model has revolutionized Natural Language Processing tasks such as Neural Machine Translation, and many efforts have been made to study the Transformer architecture, which increased its efficiency and accuracy. One potential area for improvement is to address the computation of empty tokens that the Transformer computes only to discard them later, leading to an unnecessary computational burden. To tackle this, we propose an algorithm that sorts translation sentence pairs based on their length before batching, minimizing the waste of computing power. Since the amount of sorting could violate the independent and identically distributed (i.i.d) data assumption, we sort the data partially. In experiments, we apply the proposed method to English-Korean and English-Luganda language pairs for machine translation and show that there are gains in computational time while maintaining the performance. Our method is independent of architectures, so that it can be easily integrated into any training process with flexible data lengths.

📄 PDF Abstract BibTeX arXiv:2308.08153

Code (0)

등록된 구현이 없습니다.

Tasks

Machine TranslationNMTSentenceTranslation

Methods 이 논문이 사용한 방법론

Multi-Head Attention 설명 없음

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Position-Wise Feed-Forward Layer 설명 없음

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Residual Connection 설명 없음

FS-SS: Few-Shot Learning for Fast and Accurate Spike Sorting of High-channel Count Probes

2025-03-23 · Tao Fang, Majid Zamani

There is a need for fast adaptation in spike sorting algorithms to implement brain-machine interface (BMIs) in different applications. Learning and adapting the functionality of the sorting process in real-time can signi…

Few-Shot Learning Meta-Learning Spike Sorting

Anomaly detection and classification for streaming data using PDEs

2016-08-15 · Bilal Abbasi, Jeff Calder, Adam M. Oberman

Nondominated sorting, also called Pareto Depth Analysis (PDA), is widely used in multi-objective optimization and has recently found important applications in multi-criteria anomaly detection. Recently, a partial differe…

Anomaly Detection Classification General Classification

E-Sort: Empowering End-to-end Neural Network for Multi-channel Spike Sorting with Transfer Learning and Fast Post-processing

2024-09-19 · Yuntao Han, Shiwei Wang

Decoding extracellular recordings is a crucial task in electrophysiology and brain-computer interfaces. Spike sorting, which distinguishes spikes and their putative neurons from extracellular recordings, becomes computat…

Few-Shot Learning Spike Sorting Transfer Learning

Object sorting using faster R-CNN

2020-12-29 · Pengchang Chen, Vinayak Elangovan

In a factory production line, different industry parts need to be quickly differentiated and sorted for further process. Parts can be of different colors and shapes. It is tedious for humans to differentiate and sort the…

Object

Fast Differentiable Sorting and Ranking

2020-02-20 · ICML 2020 1 · Mathieu Blondel, Olivier Teboul, Quentin Berthet, Josip Djolonga

The sorting operation is one of the most commonly used building blocks in computer programming. In machine learning, it is often used for robust statistics. However, seen as a function, it is piecewise linear and as a re…