Evaluating Byte and Wordpiece Level Models for Massively Multilingual Semantic Parsing

2022-12-14 · Massimo Nicosia, Francesco Piccinno

Token free approaches have been successfully applied to a series of word and span level tasks. In this work, we compare a byte-level (ByT5) and a wordpiece based (mT5) sequence to sequence model on the 51 languages of the MASSIVE multilingual semantic parsing dataset. We examine multiple experimental settings: (i) zero-shot, (ii) full gold data and (iii) zero-shot with synthetic data. By leveraging a state-of-the-art label projection method for machine translated examples, we are able to reduce the gap in exact match accuracy to only 5 points with respect to a model trained on gold data from all the languages. We additionally provide insights on the cross-lingual transfer of ByT5 and show how the model compares with respect to mT5 across all parameter sizes.

📄 PDF Abstract BibTeX arXiv:2212.07223

Code (0)

등록된 구현이 없습니다.

Tasks

Cross-Lingual Transfer Semantic Parsing

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Inverse Square Root Schedule Inverse Square Root is a learning rate schedule 1 / $\sqrt{\max\left(n, k\right)}$ where $n$ is the current training iteration and $k$ is the number of warm-up steps. This…

Attention Dropout Attention Dropout is a type of dropout used in attention-based architectures, where elements are randomly dropped out of the…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Multi-Head Attention 설명 없음

Residual Connection 설명 없음

Refunds@Expedia|||How do I get a full refund from Expedia? “How do I get a full refund from Expedia? How do I get a full refund from Expedia? – Call ☎️ +1-(888) 829 (0881) or +1-805-330-4056 or +1-805-330-4056 for Quick Help &…

Wine is not v i n. On the Compatibility of Tokenizations across Languages

2021-11-01 · Findings (EMNLP) 2021 11 · Antonis Maronikolakis, Philipp Dufter, Hinrich Schütze

The size of the vocabulary is a central design choice in large pretrained language models, with respect to both performance and memory requirements. Typically, subword tokenization algorithms such as byte pair encoding a…

Wine is Not v i n. -- On the Compatibility of Tokenizations Across Languages

2021-09-13 · Antonis Maronikolakis, Philipp Dufter, Hinrich Schütze

ByT5 model for massively multilingual grapheme-to-phoneme conversion

2022-04-06 · Jian Zhu, Cong Zhang, David Jurgens

In this study, we tackle massively multilingual grapheme-to-phoneme conversion through implementing G2P models based on ByT5. We have curated a G2P dataset from various sources that covers around 100 languages and traine…

Grapheme-to-Phoneme Conversion

MoCE: Adaptive Mixture of Contextualization Experts for Byte-based Neural Machine Translation

2024-11-03 · Langlin Huang, Mengyu Bu, Yang Feng

Byte-based machine translation systems have shown significant potential in massively multilingual settings. Unicode encoding, which maps each character to specific byte(s), eliminates the emergence of unknown words, even…

Machine Translation Sentence

Evaluating the Cross-Lingual Effectiveness of Massively Multilingual Neural Machine Translation

2019-09-01 · Aditya Siddhant, Melvin Johnson, Henry Tsai, Naveen Arivazhagan 외

The recently proposed massively multilingual neural machine translation (NMT) system has been shown to be capable of translating over 100 languages to and from English within a single model. Its improved translation perf…

Cross-Lingual Transfer Machine Translation NMT Transfer Learning+1