Neural Network Architecture Optimization through Submodularity and Supermodularity
Deep learning models' architectures, including depth and width, are key factors influencing models' performance, such as test accuracy and computation time. This paper solves two problems: given computation time budget, choose an architecture to maximize accuracy, and given accuracy requirement, choose an architecture to minimize computation time. We convert this architecture optimization into a subset selection problem. With accuracy's submodularity and computation time's supermodularity, we propose efficient greedy optimization algorithms. The experiments demonstrate our algorithm's ability to find more accurate models or faster models. By analyzing architecture evolution with growing time budget, we discuss relationships among accuracy, time and architecture, and give suggestions on neural network architecture design.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Unifying Formal Explanations: A Complexity-Theoretic Perspective
Previous work has explored the computational complexity of deriving two fundamental types of explanations for ML model predictions: (1) *sufficient reasons*, which are subsets of input features that, when fixed, determin…
Batch greedy maximization of non-submodular functions: Guarantees and applications to experimental design
We propose and analyze batch greedy heuristics for cardinality constrained maximization of non-submodular non-decreasing set functions. We consider the standard greedy paradigm, along with its distributed greedy and stoc…
Experimental DesignSubmodularity In Machine Learning and Artificial Intelligence
In this manuscript, we offer a gentle review of submodularity and supermodularity and their properties. We offer a plethora of submodular definitions; a full description of a number of example submodular functions and th…
Abstractive Text SummarizationBIG-bench Machine Learningfeature selectionComposite Sorting
This paper introduces an assignment model with concave costs of skill gaps, which arise generally when firms mitigate costs of mismatch as in Stigler (1939) and Laffont and Tirole (1986, 1991). Concave costs of skill gap…
Global Reinforcement Learning: Beyond Linear and Convex Rewards via Submodular Semi-gradient Methods
In classic Reinforcement Learning (RL), the agent maximizes an additive objective of the visited states, e.g., a value function. Unfortunately, objectives of this type cannot model many real-world applications such as ex…
Imitation LearningReinforcement Learning (RL)