Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation
We introduce a model for constructing vector representations of words by composing characters using bidirectional LSTMs. Relative to traditional word representation models that have independent vectors for each word type, our model requires only a single vector per character type and a fixed set of parameters for the compositional model. Despite the compactness of this model and, more importantly, the arbitrary nature of the form-function relationship in language, our "composed" word representations yield state-of-the-art results in language modeling and part-of-speech tagging. Benefits over traditional baselines are particularly pronounced in morphologically rich languages (e.g., Turkish).
Code (1)
Tasks
FormLanguage ModelingLanguage ModellingPart-Of-Speech TaggingSimilar Papers 제목 키워드 기반
ChatGPT as Data Augmentation for Compositional Generalization: A Case Study in Open Intent Detection
Open intent detection, a crucial aspect of natural language understanding, involves the identification of previously unseen intents in user-generated text. Despite the progress made in this field, challenges persist in h…
Data AugmentationIntent DetectionNatural Language UnderstandingOpen Intent DetectionLearning a Compositional Semantics for Freebase with an Open Predicate Vocabulary
We present an approach to learning a model-theoretic semantics for natural language tied to Freebase. Crucially, our approach uses an open predicate vocabulary, enabling it to produce denotations for phrases such as {``}…
Coreference ResolutionOpen Information ExtractionQuestion AnsweringSemantic Parsing+1Mehler's Formula, Branching Process, and Compositional Kernels of Deep Neural Networks
We utilize a connection between compositional kernels and branching processes via Mehler's formula to study deep neural networks. This new probabilistic insight provides us a novel perspective on the mathematical role of…
MemorizationFrom Words to Worlds: Compositionality for Cognitive Architectures
Large language models (LLMs) are very performant connectionist systems, but do they exhibit more compositionality? More importantly, is that part of why they perform so well? We present empirical analyses across four LLM…
Sublinearly Structured Deep Neural Networks Achieve Feature Learning Consistency for Compositional Functions
Over the past decade, deep neural networks (DNNs) have achieved remarkable success on complex machine-learning tasks, yet the theoretical foundations of their performance remain incomplete. From a statistical viewpoint, …
Image Classification