BitNet Distillation

2025-10-15 · Xun Wu, Shaohan Huang, Wenhui Wang, Ting Song, Li Dong, Yan Xia, Furu Wei arxiv

In this paper, we present BitNet Distillation (BitDistill), a lightweight pipeline that fine-tunes off-the-shelf full-precision LLMs (e.g., Qwen) into 1.58-bit precision (i.e., ternary weights {-1, 0, 1}) for specific downstream tasks, achieving strong task-specific performance with minimal computational cost. Specifically, BitDistill incorporates three key techniques: the SubLN module, as introduced in BitNet; multi-head attention distillation, based on MiniLM; and continual pre-training, which serves as a crucial warm-up step to mitigate the scalability issue of the performance gap between finetuned full-precision and 1.58-bit LLMs on specific tasks. Experimental results show that BitDistill achieves performance comparable to the full-precision counterpart models across model size, while enabling up to 10x memory savings and 2.65x faster inference on CPUs. Code is available at https://github.com/microsoft/BitNet.

📄 PDF Abstract BibTeX arXiv:2510.13998

Code (0)

등록된 구현이 없습니다.

1-bit AI Infra: Part 1.1, Fast and Lossless BitNet b1.58 Inference on CPUs

2024-10-21 · Jinheng Wang, Hansong Zhou, Ting Song, Shaoguang Mao 외

Recent advances in 1-bit Large Language Models (LLMs), such as BitNet and BitNet b1.58, present a promising approach to enhancing the efficiency of LLMs in terms of speed and energy consumption. These developments also e…

BitNet a4.8: 4-bit Activations for 1-bit LLMs

2024-11-07 · Hongyu Wang, Shuming Ma, Furu Wei

Recent research on the 1-bit Large Language Models (LLMs), such as BitNet b1.58, presents a promising direction for reducing the inference cost of LLMs while maintaining their performance. In this work, we introduce BitN…

Quantization

BitNet Text Embeddings

2026-06-24 · Zhen Li, Xin Huang, Liang Wang, Nan Yang 외 arxiv

LLM-based text embedders have substantially improved retrieval and semantic representation quality, but their deployment remains costly: large backbone models slow down embedding inference, while high-dimensional full-pr…

Representation Learning

Bitnet.cpp: Efficient Edge Inference for Ternary LLMs

2025-02-17 · Jinheng Wang, Hansong Zhou, Ting Song, Shijie Cao 외

The advent of 1-bit large language models (LLMs), led by BitNet b1.58, has spurred interest in ternary LLMs. Despite this, research and practical applications focusing on efficient edge inference for ternary LLMs remain …

Sparse-BitNet: 1.58-bit LLMs are Naturally Friendly to Semi-Structured Sparsity

2026-03-05 · Di Zhang, Xun Wu, Shaohan Huang, Yudong Wang 외 arxiv

Semi-structured N:M sparsity and low-bit quantization (e.g., 1.58-bit BitNet) are two promising approaches for improving the efficiency of large language models (LLMs), yet they have largely been studied in isolation. In…