paper-with-me

홈 › Papers

Supplementary Material: Implementation and Experiments for GAU-based Model

2022-05-12 · Zhenjie Liu

In February this year Google proposed a new Transformer variant called FLASH, which has a faster speed, lower VRAM footprint and better performance. This is achieved by designing a performant layer named GAU (Gated Attention Unit), which combines the Attention layer and FFN. In this paper, some implementation details are re-analyzed both theoretically and practically. We then propose a novel GAU-based model and pre-train it on a Chinese corpus. Results of the CLUE benchmark show that our model achieves a dev average score of 75.02, 1% higher than RoFormerV1 and being 45% faster, which is also competitive with RoFormerV2.

📄 PDF Abstract BibTeX arXiv:2205.05842

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

Multi-Head Attention 설명 없음
Attention 설명 없음
Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.
Adam 설명 없음
BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…
Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…
Residual Connection 설명 없음
Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Similar Papers 제목 키워드 기반

Supplementary Material for Efficient and Robust Automated Machine Learning

2015-01-01 · NIPS 2015 2015 1 · Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Tobias Springenberg 외

Supplementary Material for Efficient and Robust Automated Machine Learning

BIG-bench Machine LearningHyperparameter Optimization

Supplementary material for Uncorrected least-squares temporal difference with lambda-return

2019-11-14 · Takayuki Osogami

Here, we provide a supplementary material for Takayuki Osogami, "Uncorrected least-squares temporal difference with lambda-return," which appears in {\it Proceedings of the 34th AAAI Conference on Artificial Intelligence…

Change point detection for graphical models in the presence of missing values

2019-07-11 · Malte Londschien, Solt Kovács, Peter Bühlmann

We propose estimation methods for change points in high-dimensional covariance structures with an emphasis on challenging scenarios with missing values. We advocate three imputation like methods and investigate their imp…

Change Point DetectionImputationMissing ValuesModel Selection+2

Proofs and Supplementary Material: Unified Characterization and Precoding for Non-Stationary Channels

2022-02-03 · Zhibin Zou, Maqsood Careem, Aveek Dutta, Ngwe Thawdar

This document provides the supplementary material including a comprehensive related work, the complete proofs and extended evaluation results that support the manuscript, "Unified Characterization and Precoding for Non-S…

LEMMA

A Collective, Probabilistic Approach to Schema Mapping: Appendix

2017-02-11 · Angelika Kimmig, Alex Memory, Renee J. Miller, Lise Getoor

In this appendix we provide additional supplementary material to "A Collective, Probabilistic Approach to Schema Mapping." We include an additional extended example, supplementary experiment details, and proof for the co…