Famba-V: Fast Vision Mamba with Cross-Layer Token Fusion
Mamba and Vision Mamba (Vim) models have shown their potential as an alternative to methods based on Transformer architecture. This work introduces Fast Mamba for Vision (Famba-V), a cross-layer token fusion technique to enhance the training efficiency of Vim models. The key idea of Famba-V is to identify and fuse similar tokens across different Vim layers based on a suit of cross-layer strategies instead of simply applying token fusion uniformly across all the layers that existing works propose. We evaluate the performance of Famba-V on CIFAR-100. Our results show that Famba-V is able to enhance the training efficiency of Vim models by reducing both training time and peak memory usage during training. Moreover, the proposed cross-layer strategies allow Famba-V to deliver superior accuracy-efficiency trade-offs. These results all together demonstrate Famba-V as a promising efficiency enhancement technique for Vim models.
Code (1)
Tasks
MambaMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
DGFamba: Learning Flow Factorized State Space for Visual Domain Generalization
Domain generalization aims to learn a representation from the source domain, which can be generalized to arbitrary unseen target domains. A fundamental challenge for visual domain generalization is the domain gap caused …
Domain GeneralizationSparX: A Sparse Cross-Layer Connection Mechanism for Hierarchical Vision Mamba and Transformer Networks
Due to the capability of dynamic state space models (SSMs) in capturing long-range dependencies with linear-time computational complexity, Mamba has shown notable performance in NLP tasks. This has inspired the rapid dev…
Image ClassificationObject DetectionSemantic SegmentationCrossMambaTuning: Synergistic Spatial and Cross-Layer Adaptation for Machine Vision Compression
To reduce deployment cost and retraining overhead, adapting pretrained learned image compression (LIC) models to downstream machine vision tasks has attracted growing attention. However, existing methods typically insert…
parameter-efficient fine-tuningImage CompressionMambaGlue: Fast and Robust Local Feature Matching With Mamba
In recent years, robust matching methods using deep learning-based approaches have been actively studied and improved in computer vision tasks. However, there remains a persistent demand for both robust and fast matching…
MambaVADMamba: Exploring State Space Models for Fast Video Anomaly Detection
Video anomaly detection (VAD) methods are mostly CNN-based or Transformer-based, achieving impressive results, but the focus on detection accuracy often comes at the expense of inference speed. The emergence of state spa…
Anomaly DetectionComputational EfficiencyLong-range modelingMamba+5