Exploring Looping Effects in RNN-based Architectures
The paper investigates repetitive loops, a common problem in contemporary text generation (such as machine translation, language modelling, morphological inflection) systems. More specifically, we conduct a study on neural models with recurrent units by explicitly altering their decoder internal state. We use a task of morphological reinflection task as a proxy to study the effects of the changes. Our results show that the probability of the occurrence of repetitive loops is significantly reduced by introduction of an extra neural decoder output. The output should be specifically trained to produce gradually increasing value upon generation of each character of a given sequence. We also explored variations of the technique and found that feeding the extra output back to the decoder amplifies the positive effects.
Code (0)
등록된 구현이 없습니다.
Tasks
DecoderLanguage ModellingMachine TranslationMorphological InflectionText GenerationTranslationSimilar Papers 제목 키워드 기반
Kernel Looping: Eliminating Synchronization Boundaries for Peak Inference Performance
Token generation speed is critical to power the next wave of AI inference applications. GPUs significantly underperform during token generation due to synchronization overheads at kernel boundaries, utilizing only 21% of…
global-optimizationParcae: Scaling Laws For Stable Looped Language Models
Traditional fixed-depth architectures scale quality by increasing training FLOPs, typically through increased parameterization, at the expense of a higher memory footprint, or data. A potential alternative is looped arch…
Looped Diffusion Language Models
Masked diffusion models (MDMs) have emerged as a promising alternative to autoregressive models for language modeling, yet the effective design of transformer architectures for MDMs remains underexplored. In this paper, …
Inner Loop Inference for Pretrained Transformers: Unlocking Latent Capabilities Without Training
Deep Learning architectures, and in particular Transformers, are conventionally viewed as a composition of layers. These layers are actually often obtained as the sum of two contributions: a residual path that copies the…
Loopy: Seamless Video Loop Generation via Anchored Looping Shift of Positional Embedding
Looping videos are essential for practical applications such as web graphics, game development, and social media. However, existing approaches typically fail to generate high-quality looping videos due to the neglect of …
Video GenerationStyle Transfer