Betti numbers of attention graphs is all you really need
We apply methods of topological analysis to the attention graphs, calculated on the attention heads of the BERT model (Devlin et al. (2019)). Our research shows that the classifier built upon basic persistent topological features (namely, Betti numbers) of the trained neural network can achieve classification results on par with the conventional classification method. We show the relevance of such topological text representation on three text classification benchmarks. For the best of our knowledge, it is the first attempt to analyze the topology of an attention-based neural network, widely used for Natural Language Processing.
Code (0)
등록된 구현이 없습니다.
Tasks
AllClassificationtext-classificationText ClassificationSimilar Papers 제목 키워드 기반
Betti numbers of attention graphs is all you really need
We apply methods of topological analysis to the attention graphs, calculated on the attention heads of the BERT model ( arXiv:1810.04805v2 ). Our research shows that the classifier built upon basic persistent topological…
AllClassificationtext-classificationText ClassificationUnsupervised machine learning for physical concepts
In recent years, machine learning methods have been used to assist scientists in scientific research. Human scientific theories are based on a series of concepts. How machine learns the concepts from experimental data wi…
BIG-bench Machine LearningDive into Layers: Neural Network Capacity Bounding using Algebraic Geometry
The empirical results suggest that the learnability of a neural network is directly related to its size. To mathematically prove this, we borrow a tool in topological algebra: Betti numbers to measure the topological geo…
Using Topological Framework for the Design of Activation Function and Model Pruning in Deep Neural Networks
Success of deep neural networks in diverse tasks across domains of computer vision, speech recognition and natural language processing, has necessitated understanding the dynamics of training process and also working of …
Binary Classificationspeech-recognitionSpeech RecognitionTopological Data Analysis of Biological Aggregation Models
We apply tools from topological data analysis to two mathematical models inspired by biological aggregations such as bird flocks, fish schools, and insect swarms. Our data consists of numerical simulation output from the…
Topological Data Analysis