ConvFormer: Revisiting Transformer for Sequential User Modeling
Sequential user modeling, a critical task in personalized recommender systems, focuses on predicting the next item a user would prefer, requiring a deep understanding of user behavior sequences. Despite the remarkable success of Transformer-based models across various domains, their full potential in comprehending user behavior remains untapped. In this paper, we re-examine Transformer-like architectures aiming to advance state-of-the-art performance. We start by revisiting the core building blocks of Transformer-based methods, analyzing the effectiveness of the item-to-item mechanism within the context of sequential user modeling. After conducting a thorough experimental analysis, we identify three essential criteria for devising efficient sequential user models, which we hope will serve as practical guidelines to inspire and shape future designs. Following this, we introduce ConvFormer, a simple but powerful modification to the Transformer architecture that meets these criteria, yielding state-of-the-art results. Additionally, we present an acceleration technique to minimize the complexity associated with processing extremely long sequences. Experiments on four public datasets showcase ConvFormer's superiority and confirm the validity of our proposed criteria.
Code (0)
등록된 구현이 없습니다.
Tasks
Recommendation SystemsMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
EdgeConvFormer: Dynamic Graph CNN and Transformer based Anomaly Detection in Multivariate Time Series
Transformer-based models for anomaly detection in multivariate time series can benefit from the self-attention mechanism due to its advantage in modeling long-term dependencies. However, Transformer-based anomaly detecti…
Anomaly DetectionTime SeriesPointConvFormer: Revenge of the Point-based Convolution
We introduce PointConvFormer, a novel building block for point cloud based deep network architectures. Inspired by generalization theory, PointConvFormer combines ideas from point convolution, where filter weights are on…
Scene Flow EstimationSemantic SegmentationConvFormer: Plug-and-Play CNN-Style Transformers for Improving Medical Image Segmentation
Transformers have been extensively studied in medical image segmentation to build pairwise long-range dependence. Yet, relatively limited well-annotated medical image data makes transformers struggle to extract diverse g…
Image SegmentationMedical Image SegmentationSemantic SegmentationQ-value Regularized Decision ConvFormer for Offline Reinforcement Learning
As a data-driven paradigm, offline reinforcement learning (Offline RL) has been formulated as sequence modeling, where the Decision Transformer (DT) has demonstrated exceptional capabilities. Unlike previous reinforcemen…
D4RLOffline RLreinforcement-learningReinforcement LearningConvFormer: Combining CNN and Transformer for Medical Image Segmentation
Convolutional neural network (CNN) based methods have achieved great successes in medical image segmentation, but their capability to learn global representations is still limited due to using small effective receptive f…
Image SegmentationMedical Image SegmentationSegmentationSemantic Segmentation