Not All Lotteries Are Made Equal
The Lottery Ticket Hypothesis (LTH) states that for a reasonably sized neural network, there exists a subnetwork within the same network that yields no less performance than the dense counterpart when trained from the same initialization. To the best of our knowledge, prior work regarding the LTH has only investigated overparameterized models, and the emergence of the LTH is often attributed to the initial model being large, i.e., a dense sampling of tickets. In this blog post, we present evidence that challenges this notion of the LTH. We investigate the effect of model size and the ease of finding winning tickets. Through this work, we show that winning tickets is easier to find for smaller models.
Code (0)
등록된 구현이 없습니다.
Tasks
AllSimilar Papers 제목 키워드 기반
Not All Lotteries Are Made Equal
The Lottery Ticket Hypothesis (LTH) states that for a reasonably sized neural network, a sub-network within the same network yields no less performance than the dense counterpart when trained from the same initialization…
AllRelationTicket SearchMaking school choice lotteries transparent
Lotteries are commonly employed in school choice to fairly resolve priority ties; however, current practices leave students uninformed about their lottery outcomes when submitting preferences. This paper advocates for re…
Decision MakingRegret theory, Allais' Paradox, and Savage's omelet
We study a sufficiently general regret criterion for choosing between two probabilistic lotteries. For independent lotteries, the criterion is consistent with stochastic dominance and can be made transitive by a unique c…
The Near Miss Effect and the Framing of Lotteries
We present a framework for analyzing the near miss effect in lotteries. A decision maker (DM) facing a lottery, falsely interprets losing outcomes that are close to winning ones, as a sign that success is within reach. A…
Robust AI Evaluation through Maximal Lotteries
The standard way to evaluate language models on subjective tasks is through pairwise comparisons: an annotator chooses the "better" of two responses to a prompt. Leaderboards aggregate these comparisons into a single Bra…