A Transformer-based Neural Architecture Search Method
This paper presents a neural architecture search method based on Transformer architecture, searching cross multihead attention computation ways for different number of encoder and decoder combinations. In order to search for neural network structures with better translation results, we considered perplexity as an auxiliary evaluation metric for the algorithm in addition to BLEU scores and iteratively improved each individual neural network within the population by a multi-objective genetic algorithm. Experimental results show that the neural network structures searched by the algorithm outperform all the baseline models, and that the introduction of the auxiliary evaluation metric can find better models than considering only the BLEU score as an evaluation metric.
Code (1)
Tasks
DecoderNeural Architecture SearchMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Neural Architecture Search on Efficient Transformers and Beyond
Recently, numerous efficient Transformers have been proposed to reduce the quadratic computational complexity of standard Transformers caused by the Softmax attention. However, most of them simply swap Softmax with an ef…
Computational Efficiencyimage-classificationImage ClassificationMachine Translation+1HyTAS: A Hyperspectral Image Transformer Architecture Search Benchmark and Analysis
Hyperspectral Imaging (HSI) plays an increasingly critical role in precise vision tasks within remote sensing, capturing a wide spectrum of visual data. Transformer architectures have significantly enhanced HSI task perf…
Model DiscoveryGLiT: Neural Architecture Search for Global and Local Image Transformer
We introduce the first Neural Architecture Search (NAS) method to find a better transformer architecture for image recognition. Recently, transformers without CNN-based backbones are found to achieve impressive performan…
image-classificationImage ClassificationNeural Architecture SearchUniNet: Unified Architecture Search with Convolution, Transformer, and MLP
Recently, transformer and multi-layer perceptron (MLP) architectures have achieved impressive results on various vision tasks. A few works investigated manually combining those operators to design visual network architec…
Image Classificationobject-detectionObject DetectionSemantic SegmentationThe Evolved Transformer
Recent works have highlighted the strength of the Transformer architecture on sequence tasks while, at the same time, neural architecture search (NAS) has begun to outperform human-designed models. Our goal is to apply N…
Machine TranslationNeural Architecture Search