Exploring Kernel Functions in the Softmax Layer for Contextual Word Classification
Prominently used in support vector machines and logistic regressions, kernel functions (kernels) can implicitly map data points into high dimensional spaces and make it easier to learn complex decision boundaries. In this work, by replacing the inner product function in the softmax layer, we explore the use of kernels for contextual word classification. In order to compare the individual kernels, experiments are conducted on standard language modeling and machine translation tasks. We observe a wide range of performances across different kernel settings. Extending the results, we look at the gradient properties, investigate various mixture strategies and examine the disambiguation abilities.
Code (0)
등록된 구현이 없습니다.
Tasks
General ClassificationLanguage ModelingLanguage ModellingMachine TranslationTranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Exploring the Frontiers of Softmax: Provable Optimization, Applications in Diffusion Model, and Beyond
The softmax activation function plays a crucial role in the success of large language models (LLMs), particularly in the self-attention mechanism of the widely adopted Transformer architecture. However, the underlying le…
The Kernel Mixture Network: A Nonparametric Method for Conditional Density Estimation of Continuous Random Variables
This paper introduces the kernel mixture network, a new method for nonparametric estimation of conditional probability densities using neural networks. We model arbitrarily complex conditional densities as linear combina…
Density EstimationEfficient Contextual Representation Learning Without Softmax Layer
Contextual representation models have achieved great success in improving various downstream tasks. However, these language-model-based encoders are difficult to train due to the large parameter sizes and high computatio…
Dimensionality ReductionLanguage ModelingLanguage ModellingRepresentation Learning+1MEP: Multiple Kernel Learning Enhancing Relative Positional Encoding Length Extrapolation
When the predicted sequence length exceeds the length seen during training, the transformer's inference accuracy diminishes. Existing relative position encoding methods, such as those based on the ALiBi technique, addres…
Attention in SRAM on Tenstorrent Grayskull
When implementations of the Transformer's self-attention layer utilize SRAM instead of DRAM, they can achieve significant speedups. The Tenstorrent Grayskull architecture provides a large SRAM, distributed across a grid …
CPUGPU