Text vectorization via transformer-based language models and n-gram perplexities
As the probability (and thus perplexity) of a text is calculated based on the product of the probabilities of individual tokens, it may happen that one unlikely token significantly reduces the probability (i.e., increase the perplexity) of some otherwise highly probable input, while potentially representing a simple typographical error. Also, given that perplexity is a scalar value that refers to the entire input, information about the probability distribution within it is lost in the calculation (a relatively good text that has one unlikely token and another text in which each token is equally likely they can have the same perplexity value), especially for longer texts. As an alternative to scalar perplexity this research proposes a simple algorithm used to calculate vector values based on n-gram perplexities within the input. Such representations consider the previously mentioned aspects, and instead of a unique value, the relative perplexity of each text token is calculated, and these values are combined into a single vector representing the input.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
TransfoRNN: Capturing the Sequential Information in Self-Attention Representations for Language Modeling
In this paper, we describe the use of recurrent neural networks to capture sequential information from the self-attention representations to improve the Transformers. Although self-attention mechanism provides a means to…
Language ModelingLanguage Modellingspeech-recognitionSpeech RecognitionCharacter n-gram Embeddings to Improve RNN Language Models
This paper proposes a novel Recurrent Neural Network (RNN) language model that takes advantage of character information. We focus on character n-grams based on research in the field of word embedding construction (Wietin…
Headline GenerationLanguage ModelingLanguage ModellingMachine Translation+2Scaling Recurrent Neural Network Language Models
This paper investigates the scaling properties of Recurrent Neural Network Language Models (RNNLMs). We discuss how to train very large RNNs on GPUs and address the questions of how RNNLMs scale with respect to model siz…
Language ModellingMachine TranslationTranslationPerformance Evaluation of General Purpose Large Language Models for Basic Linear Algebra Subprograms Code Generation
Generative AI technology based on Large Language Models (LLM) has been developed and applied to assist or automatically generate program codes. In this paper, we evaluate the capability of existing general LLMs for Basic…
Code GenerationLanguage Models with Transformers
The Transformer architecture is superior to RNN-based models in computational efficiency. Recently, GPT and BERT demonstrate the efficacy of Transformer models on various NLP tasks using pre-trained language models on la…
Computational EfficiencyLanguage ModelingLanguage ModellingNeural Architecture Search