Deep Networks With Large Output Spaces
Deep neural networks have been extremely successful at various image, speech, video recognition tasks because of their ability to model deep structures within the data. However, they are still prohibitively expensive to train and apply for problems containing millions of classes in the output layer. Based on the observation that the key computation common to most neural network layers is a vector/matrix product, we propose a fast locality-sensitive hashing technique to approximate the actual dot product enabling us to scale up the training and inference to millions of output classes. We evaluate our technique on three diverse large-scale recognition tasks and show that our approach can train large-scale models at a faster rate (in terms of steps/total time) compared to baseline methods.
Code (0)
등록된 구현이 없습니다.
Tasks
Video RecognitionSimilar Papers 제목 키워드 기반
A General Framework for Consistent Structured Prediction with Implicit Loss Embeddings
We propose and analyze a novel theoretical and algorithmic framework for structured prediction. While so far the term has referred to discrete output spaces, here we consider more general settings, such as manifolds or s…
Structured PredictionPECOS: Prediction for Enormous and Correlated Output Spaces
Many large-scale applications amount to finding relevant results from an enormous output space of potential candidates. For example, finding the best matching product from a large catalog or suggesting related search phr…
PredictionLearning Representation in Colour Conversion
Colours can be represented in an infinite set of spaces highlighting distinct features. In this work, we study the structure of colour representation in variational autoencoders (VAEs) and investigate whether a specific …
image-classificationImage ClassificationScene SegmentationNeural networks in non-metric spaces
Leveraging the infinite dimensional neural network architecture we proposed in arXiv:2109.13512v4 and which can process inputs from Fr\'echet spaces, and using the universal approximation property shown therein, we now l…
Fast Label Embeddings for Extremely Large Output Spaces
Many modern multiclass and multilabel problems are characterized by increasingly large output spaces. For these problems, label embeddings have been shown to be a useful primitive that can improve computational and stati…