RankNAS: Efficient Neural Architecture Search by Pairwise Ranking
This paper addresses the efficiency challenge of Neural Architecture Search (NAS) by formulating the task as a ranking problem. Previous methods require numerous training examples to estimate the accurate performance of architectures, although the actual goal is to find the distinction between "good" and "bad" candidates. Here we do not resort to performance predictors. Instead, we propose a performance ranking method (RankNAS) via pairwise ranking. It enables efficient architecture search using much fewer training examples. Moreover, we develop an architecture selection method to prune the search space and concentrate on more promising candidates. Extensive experiments on machine translation and language modeling tasks show that RankNAS can design high-performance architectures while being orders of magnitude faster than state-of-the-art NAS systems.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingMachine TranslationNeural Architecture SearchTranslationSimilar Papers 제목 키워드 기반
An Empirical Study of Uniform-Architecture Knowledge Distillation in Document Ranking
Although BERT-based ranking models have been commonly used in commercial search engines, they are usually time-consuming for online ranking tasks. Knowledge distillation, which aims at learning a smaller model with compa…
Document RankingKnowledge DistillationRanking architectures using meta-learning
Neural architecture search has recently attracted lots of research efforts as it promises to automate the manual design of neural networks. However, it requires a large amount of computing resources and in order to allev…
Meta-LearningNeural Architecture SearchBoth Efficiency and Effectiveness! A Large Scale Pre-ranking Framework in Search System
In the realm of search systems, multi-stage cascade architecture is a prevalent method, typically consisting of sequential modules such as matching, pre-ranking, and ranking. It is generally acknowledged that the model u…
feature selectionFrom Semantic Retrieval to Pairwise Ranking: Applying Deep Learning in E-commerce Search
We introduce deep learning models to the two most important stages in product search at JD.com, one of the largest e-commerce platforms in the world. Specifically, we outline the design of a deep learning system that ret…
Deep LearningRe-RankingRetrievalSemantic RetrievalRanking Passages for Argument Convincingness
In data ranking applications, pairwise annotation is often more consistent than cardinal annotation for learning ranking models. We examine this in a case study on ranking text passages for argument convincingness. Our t…