Nearest Class-Center Simplification through Intermediate Layers
Recent advances in theoretical Deep Learning have introduced geometric properties that occur during training, past the Interpolation Threshold -- where the training error reaches zero. We inquire into the phenomena coined Neural Collapse in the intermediate layers of the networks, and emphasize the innerworkings of Nearest Class-Center Mismatch inside the deepnet. We further show that these processes occur both in vision and language model architectures. Lastly, we propose a Stochastic Variability-Simplification Loss (SVSL) that encourages better geometrical features in intermediate layers, and improves both train metrics and generalization.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingSimilar Papers 제목 키워드 기반
Exploring Learned Representations of Neural Networks with Principal Component Analysis
Understanding feature representation for deep neural networks (DNNs) remains an open question within the general field of explainable AI. We use principal component analysis (PCA) to study the performance of a k-nearest …
Approximating $(k,\ell)$-center clustering for curves
The Euclidean $k$-center problem is a classical problem that has been extensively studied in computer science. Given a set $\mathcal{G}$ of $n$ points in Euclidean space, the problem is to determine a set $\mathcal{C}$ o…
ClusteringSemantic Graphs for Syntactic Simplification: A Revisit from the Age of LLM
Symbolic sentence meaning representations, such as AMR (Abstract Meaning Representation) provide expressive and structured semantic graphs that act as intermediates that simplify downstream NLP tasks. However, the instru…
Abstract Meaning RepresentationInstruction FollowingSentenceComputer Vision-aided Atom Tracking in STEM Imaging
To address the SMC'17 data challenge -- "Data mining atomically resolved images for material properties", we first used the classic "blob detection" algorithms developed in computer vision to identify all atom centers in…
All