Data Augmentation for Transformer-based G2P

2020-07-01 · WS 2020 7 · Zach Ryan, Mans Hulden

The Transformer model has been shown to outperform other neural seq2seq models in several character-level tasks. It is unclear, however, if the Transformer would benefit as much as other seq2seq models from data augmentation strategies in the low-resource setting. In this paper we explore strategies for data augmentation in the g2p task together with the Transformer model. Our results show that a relatively simple alignment-based strategy of identifying consistent input-output subsequences in grapheme-phoneme data coupled together with a subsequent splicing together of such pieces to generate hallucinated data works well in the low-resource setting, often delivering substantial performance improvement over a standard Transformer model.

📄 PDF Abstract BibTeX

Code (0)

등록된 구현이 없습니다.

Tasks

Data Augmentation

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Position-Wise Feed-Forward Layer 설명 없음

Sigmoid Activation 설명 없음

Tanh Activation 설명 없음

LSTM An LSTM is a type of recurrent neural network that addresses the vanishing gradient problem in vanilla…

Residual Connection 설명 없음

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

How Effective is Task-Agnostic Data Augmentation for Pretrained Transformers?

2020-10-05 · Findings of the Association for Computational Linguistics 2020 · Shayne Longpre, Yu Wang, Christopher DuBois

Task-agnostic forms of data augmentation have proven widely effective in computer vision, even on pretrained models. In NLP similar results are reported most commonly for low data regimes, non-pretrained models, or situa…

Data Augmentation Translation

Leveraging the Power of Data Augmentation for Transformer-based Tracking

2023-09-15 · Jie Zhao, Johan Edstedt, Michael Felsberg, Dong Wang 외

Due to long-distance correlation and powerful pretrained models, transformer-based methods have initiated a breakthrough in visual object tracking performance. Previous works focus on designing effective architectures su…

Data Augmentation Object Tracking Visual Object Tracking

Exploring Token-Level Augmentation in Vision Transformer for Semi-Supervised Semantic Segmentation

2025-03-04 · Dengke Zhang, Quan Tang, Fagui Liu, C. L. Philip Chen 외

Semi-supervised semantic segmentation has witnessed remarkable advancements in recent years. However, existing algorithms are based on convolutional neural networks and directly applying them to Vision Transformers poses…

Data Augmentation Diversity Image Augmentation Semantic Segmentation+1

TokenMixup: Efficient Attention-guided Token-level Data Augmentation for Transformers

2022-10-14 · Hyeong Kyu Choi, Joonmyung Choi, Hyunwoo J. Kim

Mixup is a commonly adopted data augmentation technique for image classification. Recent advances in mixup methods primarily focus on mixing based on saliency. However, many saliency detectors require intense computation…

Data Augmentation image-classification Image Classification

Mitigating Degree Biases in Message Passing Mechanism by Utilizing Community Structures

2023-12-28 · Van Thuy Hoang, O-Joun Lee

This study utilizes community structures to address node degree biases in message-passing (MP) via learnable graph augmentations and novel graph transformers. Recent augmentation-based methods showed that MP neural netwo…

Node Classification Node Clustering Self-Supervised Learning