홈 › Papers

Efficient Context Scaling with LongCat ZigZag Attention

2025-12-30 · Chen Zhang, Yang Bai, Jiahuan Li, Anchun Gui, Keheng Wang, Feifan Liu, Guanyu Wu, Yuwei Jiang, Defei Bu, Li Wei, Haihang Jing, Hongyin Tang, Xin Chen, Xiangzhou Huang, Fengcun Li, Rongxiang Weng, Yulei Qian, Yifan Lu, Yerui Sun, Jingang Wang, Yuchen Xie, Xunliang Cai arxiv

We introduce LongCat ZigZag Attention (LoZA), which is a sparse attention scheme designed to transform any existing full-attention models into sparse versions with rather limited compute budget. In long-context scenarios, LoZA can achieve significant speed-ups both for prefill-intensive (e.g., retrieval-augmented generation) and decode-intensive (e.g., tool-integrated reasoning) cases. Specifically, by applying LoZA to LongCat-Flash during mid-training, we serve LongCat-Flash-Exp as a long-context foundation model that can swiftly process up to 1 million tokens, enabling efficient long-term reasoning and long-horizon agentic capabilities.

📄 PDF Abstract BibTeX arXiv:2512.23966

Code (0)

등록된 구현이 없습니다.

LongCat Sparse Attention: Taming the Lightning via Streaming-aware Hierarchical Cross-Layer Indexing

2026-08-03 · Wen Zan, Jiaqi Zhang, Jianchao Tan, Hong Liu 외 arxiv

DeepSeek Sparse Attention (DSA) enables efficient long-context modeling through its Lightning Indexer. However, practical deployment remains constrained by the indexer's expensive $O(L^2)$ scoring overhead and the hardwa…

LongCat-Flash Technical Report

2025-09-01 · Meituan LongCat Team, Bayan, Bei Li, Bingye Lei 외 arxiv

We introduce LongCat-Flash, a 560-billion-parameter Mixture-of-Experts (MoE) language model designed for both computational efficiency and advanced agentic capabilities. Stemming from the need for scalable efficiency, Lo…

Computational Efficiency

Scaling Embeddings Outperforms Scaling Experts in Language Models

2026-01-29 · Hong Liu, Jiaqi Zhang, Chao Wang, Xing Hu 외 arxiv

While Mixture-of-Experts (MoE) architectures have become the standard for sparsity scaling in large language models, they increasingly face diminishing returns and system-level bottlenecks. In this work, we explore embed…

Online Draft Co-Training for Speculative Decoding in Large-Scale, Long-Context RL Post-Training

2026-09-07 · Zili Wang, Zhaopeng Qiu, Yuekai Zhang, Shuang Yu 외 hf

Speculative decoding accelerates rollout generation, which dominates the cost of reinforcement learning (RL) post-training. Online co-training can further increase the draft's accuracy, yielding greater speedups. However…

Reinforcement Learning

LongCat-Video Technical Report

2025-10-25 · Meituan LongCat Team, Xunliang Cai, Qilong Huang, Zhuoliang Kang 외 arxiv

Video generation is a critical pathway toward world models, with efficient long video inference as a key capability. Toward this end, we introduce LongCat-Video, a foundational video generation model with 13.6B parameter…

Video Generation