CodeSSM: Towards State Space Models for Code Understanding
Although transformers are widely used for various code-specific tasks, they have some significant limitations. In this paper, we investigate State Space Models (SSMs) as a potential alternative to transformers for code understanding tasks, such as code retrieval, classification, and clone detection. Previous research has already demonstrated that SSMs are more compute-efficient than transformers. In our work, we show that SSMs are also more sample-efficient and can effectively extrapolate to longer contexts (beyond the pretraining context) during fine-tuning. Through comprehensive experiments, we demonstrate that SSMs could serve as a viable alternative to transformers for code understanding tasks, while addressing some of the major limitations associated with transformers.
Code (0)
등록된 구현이 없습니다.
Tasks
Clone DetectionLanguage ModelingLanguage ModellingMasked Language ModelingmodelRetrievalState Space ModelsSimilar Papers 제목 키워드 기반
Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generation
Unified multimodal models typically rely on pretrained vision encoders and use separate visual representations for understanding and generation, creating misalignment between the two tasks and preventing fully end-to-end…
Image GenerationPCAAE: Principal Component Analysis Autoencoder for organising the latent space of generative networks
Autoencoders and generative models produce some of the most spectacular deep learning results to date. However, understanding and controlling the latent space of these models presents a considerable challenge. Drawing in…
TUNA: Taming Unified Visual Representations for Native Unified Multimodal Models
Unified multimodal models (UMMs) aim to jointly perform multimodal understanding and generation within a single framework. We present TUNA, a native UMM that builds a unified continuous visual representation by cascading…
Video GenerationImage EditingNeural Architecture Search for Natural Language Understanding
Neural architecture search (NAS) has made rapid progress incomputervision,wherebynewstate-of-the-artresultshave beenachievedinaseriesoftaskswithautomaticallysearched neural network (NN) architectures. In contrast, NAS ha…
AvgLanguage ModelingLanguage ModellingNatural Language Understanding+1Towards Understanding What State Space Models Learn About Code
State Space Models (SSMs) have emerged as an efficient alternative to the Transformer architecture. Prior work shows that, when trained under comparable conditions, SSMs can match or surpass Transformers on code understa…