Modern Methods for Text Generation

2020-09-10 · Dimas Munoz Montesinos

Synthetic text generation is challenging and has limited success. Recently, a new architecture, called Transformers, allow machine learning models to understand better sequential data, such as translation or summarization. BERT and GPT-2, using Transformers in their cores, have shown a great performance in tasks such as text classification, translation and NLI tasks. In this article, we analyse both algorithms and compare their output quality in text generation tasks.

📄 PDF Abstract BibTeX arXiv:2009.04968

Code (2)

DimasDMM/nlp-completer pytorch

DimasDMM/transformers pytorch

Tasks

BIG-bench Machine Learning text-classification Text Classification Text Generation Translation

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Cosine Annealing Cosine Annealing is a type of learning rate schedule that has the effect of starting with a large learning rate that is relatively rapidly decreased to a minimum value before…

Linear Warmup With Cosine Annealing Linear Warmup With Cosine Annealing is a learning rate schedule where we increase the learning rate linearly for $n$ updates and then anneal according to a cosine schedule…

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Discriminative Fine-Tuning Discriminative Fine-Tuning is a fine-tuning strategy that is used for ULMFiT type models. Instead of using the same learning rate…

GPT-2 GPT-2 is a Transformer architecture that was notable for its size (1.5 billion parameters) on its release. The…

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Weight Decay 설명 없음

Russian Natural Language Generation: Creation of a Language Modelling Dataset and Evaluation with Modern Neural Architectures

2020-05-05 · Zein Shaheen, Gerhard Wohlgenannt, Bassel Zaity, Dmitry Mouromtsev 외

Generating coherent, grammatically correct, and meaningful text is very challenging, however, it is crucial to many modern NLP systems. So far, research has mostly focused on English language, for other languages both st…

Diversity Language Modeling Language Modelling Text Generation

Generative AI-Based Text Generation Methods Using Pre-Trained GPT-2 Model

2024-04-02 · Rohit Pandey, Hetvi Waghela, Sneha Rakshit, Aparna Rangari 외

This work delved into the realm of automatic text generation, exploring a variety of techniques ranging from traditional deterministic approaches to more modern stochastic methods. Through analysis of greedy search, beam…

Text Generation

Text Generation with Speech Synthesis for ASR Data Augmentation

2023-05-22 · Zhuangqun Huang, Gil Keren, Ziran Jiang, Shashank Jain 외

Aiming at reducing the reliance on expensive human annotations, data synthesis for Automatic Speech Recognition (ASR) has remained an active area of research. While prior work mainly focuses on synthetic speech generatio…

Automatic Speech Recognition Automatic Speech Recognition (ASR)Data Augmentation speech-recognition+6

Impact and influence of modern AI in metadata management

2025-01-28 · Wenli Yang, Rui Fu, Muhammad Bilal Amin, Byeong Kang

Metadata management plays a critical role in data governance, resource discovery, and decision-making in the data-driven era. While traditional metadata approaches have primarily focused on organization, classification, …

Decision Making Management

Generation of Chinese classical poetry based on pre-trained model

2022-11-04 · Ziyao Wang, Lujin Guan, Guanyu Liu

In order to test whether artificial intelligence can create qualified classical poetry like humans, the author proposes a study of Chinese classical poetry generation based on a pre-trained model. This paper mainly tries…