Kernelized Bayesian Softmax for Text Generation
Neural models for text generation require a softmax layer with proper token embeddings during the decoding phase. Most existing approaches adopt single point embedding for each token. However, a word may have multiple senses according to different context, some of which might be distinct. In this paper, we propose KerBS, a novel approach for learning better embeddings for text generation. KerBS embodies two advantages: (a) it employs a Bayesian composition of embeddings for words with multiple senses; (b) it is adaptive to semantic variances of words and robust to rare sentence context by imposing learned kernels to capture the closeness of words (senses) in the embedding space. Empirical studies show that KerBS significantly boosts the performance of several text generation tasks.
Code (1)
Tasks
SentenceText GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Kernelized Classification in Deep Networks
We propose a kernelized classification layer for deep networks. Although conventional deep networks introduce an abundance of nonlinearity for representation (feature) learning, they almost universally use a linear class…
ClassificationGeneral ClassificationRevisiting Logistic-softmax Likelihood in Bayesian Meta-Learning for Few-Shot Classification
Meta-learning has demonstrated promising results in few-shot classification (FSC) by learning to solve new problems using prior knowledge. Bayesian methods are effective at characterizing uncertainty in FSC, which is cru…
Data AugmentationMeta-LearningStable, Fast and Accurate: Kernelized Attention with Relative Positional Encoding
The attention module, which is a crucial component in Transformer, cannot scale efficiently to long sequences due to its quadratic complexity. Many works focus on approximating the dot-then-exponentiate softmax function …
Transformers Can Implement Preconditioned Richardson Iteration for In-Context Gaussian Kernel Regression
Mechanistic accounts of in-context learning (ICL) have identified iterative algorithms for linear regression and related linear prediction tasks, often using linear or ReLU attention variants. For nonlinear ICL, prior wo…
Bayesian Sparse Factor Analysis with Kernelized Observations
Multi-view problems can be faced with latent variable models since they are able to find low-dimensional projections that fairly capture the correlations among the multiple views that characterise each datum. On the othe…
feature selectionGaussian ProcessesMULTI-VIEW LEARNING