Reg4Pru: Regularisation Through Random Token Routing for Token Pruning

2026-02-02 · Julian Wyatt, Ronald Clark, Irina Voiculescu arxiv

Transformers are widely adopted in modern vision models due to their strong ability to scale with dataset size and generalisability. However, this comes with a major drawback: computation scales quadratically to the total number of tokens. Numerous methods have been proposed to mitigate this. For example, we consider token pruning with reactivating tokens from preserved representations, but the increased computational efficiency of this method results in decreased stability from the preserved representations, leading to poorer dense prediction performance at deeper layers. In this work, we introduce Reg4Pru, a training regularisation technique that mitigates token-pruning performance loss for segmentation. We compare our models on the FIVES blood vessel segmentation dataset and find that Reg4Pru improves average precision by an absolute 46% compared to the same model trained without routing. This increase is observed using a configuration that achieves a 29% relative speedup in wall-clock time compared to the non-pruned baseline. These findings indicate that Reg4Pru is a valuable regulariser for token reduction strategies.

📄 PDF Abstract BibTeX arXiv:2602.02163

Code (0)

등록된 구현이 없습니다.

Tasks

Computational Efficiency

Adaptive Computation Depth via Learned Token Routing in Transformers

2026-04-18 · Ahmed Abdelmuniem Abdalla Mohammed arxiv

Standard transformer architectures apply the same number of layers to every token regardless of contextual difficulty. We present Token-Selective Attention (TSA), a learned per-token gate on residual updates between cons…

Layer-wise MoE Routing Locality under Shared-Prefix Code Generation: Token-Identity Decomposition and Compile-Equivalent Fork Redundancy

2026-04-19 · Shun-ichiro Hayashi, Daichi Mukunoki, Tetsuya Hoshino, Takahiro Katagiri arxiv

In LLM-based code generation, multiple code candidates are often generated in parallel from the same prompt -- for example, in best-of-N sampling or multi-candidate code completion. These requests can share KV caches thr…

Code Generation Code Completion

Less Tokens, Better Forecasts: Sparse Residual Routing for Efficient Weather Prediction

2026-07-02 · Janet Wang, Yunbei Zhang, Lin Zhao, Xi Xiao 외 arxiv

Existing ViT-based weather forecasting models apply uniform computation across all spatial tokens, even though nearby atmospheric grid points often contain similar values and large regions evolve smoothly over time. This…

Weather Forecasting

Sensitivity-Aware Thresholding and Token Routing for Activation Sparsification in Large Language Models

2026-07-09 · Bishmoy Paul, Youngmin Yi, Hoeseok Yang arxiv

Efficient inference in Large Language Models (LLMs) requires deciding where computation can be reduced while preserving model quality. We study this problem through multilayer perceptron (MLP) activation sparsification a…

Towards an empirical understanding of MoE design choices

2024-02-20 · Dongyang Fan, Bettina Messmer, Martin Jaggi

In this study, we systematically evaluate the impact of common design choices in Mixture of Experts (MoEs) on validation performance, uncovering distinct influences at token and sequence levels. We also present empirical…

Mixture-of-Experts