Echo State Neural Machine Translation
We present neural machine translation (NMT) models inspired by echo state network (ESN), named Echo State NMT (ESNMT), in which the encoder and decoder layer weights are randomly generated then fixed throughout training. We show that even with this extremely simple model construction and training procedure, ESNMT can already reach 70-80% quality of fully trainable baselines. We examine how spectral radius of the reservoir, a key quantity that characterizes the model, determines the model behavior. Our findings indicate that randomized networks can work well even for complicated sequence-to-sequence prediction NLP tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
DecoderMachine TranslationNMTTranslationSimilar Papers 제목 키워드 기반
Echo-Attention: Attend Once and Get $N$ Attentions for Free
This paper proposes echo-attention layers, an efficient method for improving the expressiveness of the self-attention layers without incurring significant parameter or training time costs. The key idea is to iteratively …
Language ModelingLanguage ModellingMachine TranslationQuestion Answering+1A Machine Learning Case Study for AI-empowered echocardiography of Intensive Care Unit Patients in low- and middle-income countries
We present a Machine Learning (ML) study case to illustrate the challenges of clinical translation for a real-time AI-empowered echocardiography system with data of ICU patients in LMICs. Such ML case study includes data…
DiversityModel SelectionEcho: Compiler-based GPU Memory Footprint Reduction for LSTM RNN Training
The Long-Short-Term-Memory Recurrent Neural Networks (LSTM RNNs) are a popular class of machine learning models for analyzing sequential data. Their training on modern GPUs, however, is limited by the GPU memory capacity…
GPUMachine TranslationNMTA Domain Translation Framework with an Adversarial Denoising Diffusion Model to Generate Synthetic Datasets of Echocardiography Images
Currently, medical image domain translation operations show a high demand from researchers and clinicians. Amongst other capabilities, this task allows the generation of new medical images with sufficiently high image qu…
DenoisingSSIMTranslationFDNet: A Deep Learning Approach with Two Parallel Cross Encoding Pathways for Precipitation Nowcasting
With the goal of predicting the future rainfall intensity in a local region over a relatively short period time, precipitation nowcasting has been a long-time scientific challenge with great social and economic impact. T…
Optical Flow Estimation