Ouroboros: On Accelerating Training of Transformer-Based Language Models

2019-09-14 · NeurIPS 2019 12 · Qian Yang, Zhouyuan Huo, Wenlin Wang, Heng Huang, Lawrence Carin

Language models are essential for natural language processing (NLP) tasks, such as machine translation and text summarization. Remarkable performance has been demonstrated recently across many NLP domains via a Transformer-based language model with over a billion parameters, verifying the benefits of model size. Model parallelism is required if a model is too large to fit in a single computing device. Current methods for model parallelism either suffer from backward locking in backpropagation or are not applicable to language models. We propose the first model-parallel algorithm that speeds the training of Transformer-based language models. We also prove that our proposed algorithm is guaranteed to converge to critical points for non-convex problems. Extensive experiments on Transformer and Transformer-XL language models demonstrate that the proposed algorithm obtains a much faster speedup beyond data parallelism, with comparable or better accuracy. Code to reproduce experiments is to be found at \url{https://github.com/LaraQianYang/Ouroboros}.

📄 PDF Abstract BibTeX arXiv:1909.06695

Code (1)

LaraQianYang/Ouroboros 공식 구현

Tasks

Language ModelingLanguage Modelling Machine Translation Text Summarization Translation

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Position-Wise Feed-Forward Layer 설명 없음

Cosine Annealing Cosine Annealing is a type of learning rate schedule that has the effect of starting with a large learning rate that is relatively rapidly decreased to a minimum value before…

Variational Dropout Variational Dropout is a regularization technique based on dropout, but uses a variational inference grounded approach. In…

Residual Connection 설명 없음

Adaptive Input Representations Adaptive Input Embeddings extend the adaptive softmax to input word representations. The factorization assigns more…

Adaptive Softmax Adaptive Softmax is a speedup technique for the computation of probability distributions over words. The adaptive softmax is…

Ouroboros: Dynamic Weight Generation for Recursive Transformers via Input-Conditioned LoRA Modulation

2026-04-02 · Jaber Jaber, Osama Jaber arxiv

Recursive transformers reuse a shared weight block across multiple depth steps, trading parameters for compute. A core limitation: every step applies the same transformation, preventing the model from composing distinct …

Ouroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decoding

2024-02-21 · Weilin Zhao, Yuxiang Huang, Xu Han, Wang Xu 외

Speculative decoding is a widely used method that accelerates the generation process of large language models (LLMs) with no compromise in model performance. It achieves this goal by using an existing smaller model for d…

Text Generation

Ouroboros: Single-step Diffusion Models for Cycle-consistent Forward and Inverse Rendering

2025-08-20 · Shanlin Sun, Yifan Wang, Hanwen Zhang, Yifeng Xiong 외 arxiv

While multi-step diffusion models have advanced both forward and inverse rendering, existing approaches often treat these problems independently, leading to cycle inconsistency and slow inference speed. In this work, we …

Inverse Rendering

Ouroboros-Spatial: Closing the Data-Model Loop for Spatial Reasoning

2026-06-10 · Enhan Zhao, Wei Wu, Yuanrui Zhang, Xueliang Zhao 외 arxiv

Spatial reasoning remains a persistent challenge for multimodal large language models (MLLMs). Existing approaches largely rely on large-scale, statically curated datasets, where all training samples are treated uniforml…

Spatial Reasoning

Ouroboros3D: Image-to-3D Generation via 3D-aware Recursive Diffusion

2024-06-05 · CVPR 2025 1 · Hao Wen, Zehuan Huang, Yaohui Wang, Xinyuan Chen 외

Existing single image-to-3D creation methods typically involve a two-stage process, first generating multi-view images, and then using these images for 3D reconstruction. However, training these two stages separately lea…

3D Generation 3D Reconstruction Denoising Image Generation+1