KERMIT: Complementing Transformer Architectures with Encoders of Explicit Syntactic Interpretations
Syntactic parsers have dominated natural language understanding for decades. Yet, their syntactic interpretations are losing centrality in downstream tasks due to the success of large-scale textual representation learners. In this paper, we propose KERMIT (Kernel-inspired Encoder with Recursive Mechanism for Interpretable Trees) to embed symbolic syntactic parse trees into artificial neural networks and to visualize how syntax is used in inference. We experimented with KERMIT paired with two state-of-the-art transformer-based universal sentence encoders (BERT and XLNet) and we showed that KERMIT can indeed boost their performance by effectively embedding human-coded universal syntactic representations in neural networks
Code (1)
Tasks
Representation LearningSentenceSyntax RepresentationText ClassificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
KERMIT - A Transformer-Based Approach for Knowledge Graph Matching
One of the strongest signals for automated matching of knowledge graphs and ontologies are textual concept descriptions. With the rise of transformer-based language models, text comparison based on meaning (rather than l…
Graph MatchingKnowledge GraphsSentenceAutonomic Architecture for Big Data Performance Optimization
The big data software stack based on Apache Spark and Hadoop has become mission critical in many enterprises. Performance of Spark and Hadoop jobs depends on a large number of configuration settings. Manual tuning is exp…
KERMIT: Generative Insertion-Based Modeling for Sequences
We present KERMIT, a simple insertion-based approach to generative modeling for sequences and sequence pairs. KERMIT models the joint distribution and its decompositions (i.e., marginals and conditionals) using a single …
Machine TranslationQuestion AnsweringRepresentation LearningTranslationContextual Embeddings for Arabic-English Code-Switched Data
Globalization has caused the rise of the code-switching phenomenon among multilingual societies. In Arab countries, code-switching between Arabic and English has become frequent, especially through social media platforms…
named-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)Question Answering+1Mechanistic Interpretability for Transformer-based Time Series Classification
Transformer-based models have become state-of-the-art tools in various machine learning tasks, including time series classification, yet their complexity makes understanding their internal decision-making challenging. Ex…
Time Series Classification