Two Losses Are Better Than One: Faster Optimization Using a Cheaper Proxy
We present an algorithm for minimizing an objective with hard-to-compute gradients by using a related, easier-to-access function as a proxy. Our algorithm is based on approximate proximal point iterations on the proxy combined with relatively few stochastic gradients from the objective. When the difference between the objective and the proxy is $\delta$-smooth, our algorithm guarantees convergence at a rate matching stochastic gradient descent on a $\delta$-smooth objective, which can lead to substantially better sample efficiency. Our algorithm has many potential applications in machine learning, and provides a principled means of leveraging synthetic data, physics simulators, mixed public and private data, and more.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
As Transfer Learning from large-scale pre-trained models becomes more prevalent in Natural Language Processing (NLP), operating these large models in on-the-edge and/or under constrained computational training or inferen…
Hate Speech DetectionKnowledge DistillationLanguage ModelingLanguage Modelling+7Stochastic Nonconvex Optimization with Large Minibatches
We study stochastic optimization of nonconvex loss functions, which are typical objectives for training neural networks. We propose stochastic approximation algorithms which optimize a series of regularized, nonlinearize…
Stochastic OptimizationLayer 2 be or Layer not 2 be: Scaling on Uniswap v3
This paper studies the market structure impact of cheaper and faster chains on the Uniswap v3 Protocol. The Uniswap Protocol is the largest decentralized application on Ethereum by both gas and blockspace used, and user …
AuroraEdge-V-2B: A Faster And Stronger Edge Visual Large Language Model
Recently, due to the advancement of multimodal technology, people are attempting to use visual large language models (VLLMs) in industrial production. Many deep learning models (DLMs) deployed in the production environme…
Better and Faster: Exponential Loss for Image Patch Matching
Recent studies on image patch matching are paying more attention on hard sample learning, because easy samples do not contribute much to the network optimization. They have proposed various hard negative sample mining st…
Image RetrievalPatch MatchingRetrievalTriplet