The Global Landscape of Neural Networks: An Overview
One of the major concerns for neural network training is that the non-convexity of the associated loss functions may cause bad landscape. The recent success of neural networks suggests that their loss landscape is not too bad, but what specific results do we know about the landscape? In this article, we review recent findings and results on the global landscape of neural networks. First, we point out that wide neural nets may have sub-optimal local minima under certain assumptions. Second, we discuss a few rigorous results on the geometric properties of wide networks such as "no bad basin", and some modifications that eliminate sub-optimal local minima and/or decreasing paths to infinity. Third, we discuss visualization and empirical explorations of the landscape for practical neural nets. Finally, we briefly discuss some convergence results and their relation to landscape results.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Global epistasis on fitness landscapes
Epistatic interactions between mutations add substantial complexity to adaptive landscapes, and are often thought of as detrimental to our ability to predict evolution. Yet, patterns of global epistasis, in which the fit…
Proverbs or Pythian Oracles? Sentiments and Emotions in Greek Sayings
Proverbs are among the most fascinating language phenomena that transcend cultural and linguistic boundaries. Yet, much of the global landscape of proverbs remains underexplored, as many cultures preserve their tradition…
A Survey of Large Language Models in Psychotherapy: Current Landscape and Future Directions
Mental health remains a critical global challenge, with increasing demand for accessible, effective interventions. Large language models (LLMs) offer promising solutions in psychotherapy by enhancing the assessment, diag…
SurveyLocal Optima Networks: A New Model of Combinatorial Fitness Landscapes
This chapter overviews a recently introduced network-based model of combinatorial landscapes: Local Optima Networks (LON). The model compresses the information given by the whole search space into a smaller mathematical …
Heuristic SearchA Review of Transformer-Based Models for Computer Vision Tasks: Capturing Global Context and Spatial Relationships
Transformer-based models have transformed the landscape of natural language processing (NLP) and are increasingly applied to computer vision tasks with remarkable success. These models, renowned for their ability to capt…
image-classificationImage Classificationobject-detectionObject Detection