홈 › Papers

Liger Kernel: Efficient Triton Kernels for LLM Training

2024-10-14 · Pin-Lun Hsu, Yun Dai, Vignesh Kothapalli, Qingquan Song, Shao Tang, Siyu Zhu, Steven Shimizu, Shivam Sahni, Haowen Ning, Yanning Chen

Training Large Language Models (LLMs) efficiently at scale presents a formidable challenge, driven by their ever-increasing computational demands and the need for enhanced performance. In this work, we introduce Liger-Kernel, an open-sourced set of Triton kernels developed specifically for LLM training. With kernel optimization techniques like kernel operation fusing and input chunking, our kernels achieve on average a 20% increase in training throughput and a 60% reduction in GPU memory usage for popular LLMs compared to HuggingFace implementations. In addition, Liger-Kernel is designed with modularity, accessibility, and adaptability in mind, catering to both casual and expert users. Comprehensive benchmarks and integration tests are built in to ensure compatibility, performance, correctness, and convergence across diverse computing environments and model architectures. The source code is available under a permissive license at: github.com/linkedin/Liger-Kernel.

📄 PDF Abstract BibTeX arXiv:2410.10989

Code (1)

linkedin/liger-kernel 공식 구현 pytorch

Tasks

ChunkingGPU

Methods 이 논문이 사용한 방법론

SET Dynamic Sparse Training method where weight mask is updated randomly periodically

DRTriton: Large-Scale Synthetic Data Driven Reinforcement Learning for Triton Kernel Generation

2026-03-23 · Siqi Guo, Ming Lin, Tianbao Yang arxiv

Developing efficient CUDA kernels is a fundamental yet challenging task in the generative AI industry. Recent research leverages Large Language Models (LLMs) to automatically convert PyTorch reference implementations to …

Reinforcement Learning

TritonRL: Training LLMs to Think and Code Triton Without Cheating

2025-10-18 · Jiin Woo, Shaowei Zhu, Allen Nie, Zhen Jia 외 arxiv

The rapid evolution of Large Language Models (LLMs) has driven a growing demand for automated, high-performance system kernels to accelerate machine learning workloads. We introduce TritonRL, a domain-specialized 8B-scal…

Reinforcement Learning Domain Adaptation

Geak: Introducing Triton Kernel AI Agent & Evaluation Benchmarks

2025-07-31 · Jianghui Wang, Vinay Joshi, Saptarshi Majumder, Xu Chao 외 arxiv

The demand for AI-generated GPU kernels is rapidly growing, influenced by the need for scalable, hardware-optimized solutions in both industry and academia. As deep learning workloads grow in complexity and diversity, it…

Code Generation

Hexagon-MLIR: An AI Compilation Stack For Qualcomm's Neural Processing Units (NPUs)

2026-02-23 · Mohammed Javed Absar, Muthu Baskaran, Abhikrant Sharma, Abhilash Bhandari 외 arxiv

In this paper, we present Hexagon-MLIR,an open-source compilation stack that targets Qualcomm Hexagon Neural Processing Unit (NPU) and provides unified support for lowering Triton kernels and PyTorch models . Built using…

rl-triton: High-Performance Triton GPU Kernels for Reinforcement Learning Credit Assignment

2026-08-18 · Lars Simon Zehnder arxiv

We present rl-triton, an open-source library of high-performance GPU kernels for reinforcement learning credit assignment, implemented in Triton. The core contribution is a unified associative scan framework that recasts…