Block-Based Double Decoders
Encoder-decoder models offer substantial inference-time savings over decoder-only models, but their pretraining objectives suffer from sparse supervision and dynamic sequence lengths, keeping them out of practice at scale. We propose block-based double decoders, a novel transformer architecture that utilizes doubly-causal block-based attention masks to train with full loss supervision and static sequence packing, combining decoder-only training efficiency with encoder-decoder inference efficiency. In scaling law experiments, block-based double decoders strongly outperform encoder-decoders and closely track decoder-only models across scales. At inference time, they cut KV-cache memory and per-token compute by at least 2/3 without sacrificing prefill caching or other existing inference optimizations available to decoder-only models.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
How to Mask in Error Correction Code Transformer: Systematic and Double Masking
In communication and storage systems, error correction codes (ECCs) are pivotal in ensuring data reliability. As deep learning's applicability has broadened across diverse domains, there is a growing research focus on ne…
DooDLeNet: Double DeepLab Enhanced Feature Fusion for Thermal-color Semantic Segmentation
In this paper we present a new approach for feature fusion between RGB and LWIR Thermal images for the task of semantic segmentation for driving perception. We propose DooDLeNet, a double DeepLab architecture with specia…
DecoderSegmentationSemantic SegmentationThermal Image SegmentationMIST: A Novel Training Strategy for Low-latencyScalable Neural Net Decoders
In this paper, we propose a low latency, robust and scalable neural net based decoder for convolutional and low-density parity-check (LPDC) coding schemes. The proposed decoders are demonstrated to have bit error rate (B…
DecoderOn the Design and Performance of Machine Learning Based Error Correcting Decoders
This paper analyzes the design and competitiveness of four neural network (NN) architectures recently proposed as decoders for forward error correction (FEC) codes. We first consider the so-called single-label neural net…
Action-List Reinforcement Learning Syndrome Decoding for Binary Linear Block Codes
This paper explores the application of reinforcement learning techniques to enhance the performance of decoding of linear block codes based on flipping bits and finding optimal decisions. We describe the methodology for …
Reinforcement Learning