TriMLP: Revenge of a MLP-like Architecture in Sequential Recommendation

2023-05-24 · Yiheng Jiang, Yuanbo Xu, Yongjian Yang, Funing Yang, Pengyang Wang, Hui Xiong

In this paper, we present a MLP-like architecture for sequential recommendation, namely TriMLP, with a novel Triangular Mixer for cross-token communications. In designing Triangular Mixer, we simplify the cross-token operation in MLP as the basic matrix multiplication, and drop the lower-triangle neurons of the weight matrix to block the anti-chronological order connections from future tokens. Accordingly, the information leakage issue can be remedied and the prediction capability of MLP can be fully excavated under the standard auto-regressive mode. Take a step further, the mixer serially alternates two delicate MLPs with triangular shape, tagged as global and local mixing, to separately capture the long range dependencies and local patterns on fine-grained level, i.e., long and short-term preferences. Empirical study on 12 datasets of different scales (50K\textasciitilde 10M user-item interactions) from 4 benchmarks (Amazon, MovieLens, Tenrec and LBSN) show that TriMLP consistently attains promising accuracy/efficiency trade-off, where the average performance boost against several state-of-the-art baselines achieves up to 14.88% with 8.65% less inference cost.

📄 PDF Abstract BibTeX arXiv:2305.14675

Code (1)

jiangyiheng1/trimlp 공식 구현 pytorch

Tasks

Sequential Recommendation

Methods 이 논문이 사용한 방법론

Multi-Head Attention 설명 없음

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Determinantal Point Process Likelihoods for Sequential Recommendation

2022-04-25 · Yuli Liu, Christian Walder, Lexing Xie

Sequential recommendation is a popular task in academic research and close to real-world application scenarios, where the goal is to predict the next action(s) of the user based on his/her previous sequence of actions. I…

Diversity Recommendation Systems Sequential Recommendation

PTF-FSR: A Parameter Transmission-Free Federated Sequential Recommender System

2024-06-08 · Wei Yuan, Chaoqun Yang, Liang Qu, Quoc Viet Hung Nguyen 외

Sequential recommender systems have made significant progress. Recently, due to increasing concerns about user data privacy, some researchers have implemented federated learning for sequential recommendation, a.k.a., Fed…

Federated Learning Language Modelling Large Language Model Recommendation Systems+1

Learning Montezuma's Revenge from a Single Demonstration

2018-12-08 · Tim Salimans, Richard Chen

We propose a new method for learning from a single demonstration to solve hard exploration tasks like the Atari game Montezuma's Revenge. Instead of imitating human demonstrations, as proposed in other recent works, our …

Montezuma's Revenge reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Expert-augmented actor-critic for ViZDoom and Montezumas Revenge

2018-09-10 · Michał Garmulewicz, Henryk Michalewski, Piotr Miłoś

We propose an expert-augmented actor-critic algorithm, which we evaluate on two environments with sparse rewards: Montezumas Revenge and a demanding maze from the ViZDoom suite. In the case of Montezumas Revenge, an agen…

On Bonus Based Exploration Methods In The Arcade Learning Environment

2020-01-01 · ICLR 2020 1 · Adrien Ali Taiga, William Fedus, Marlos C. Machado, Aaron Courville 외

Research on exploration in reinforcement learning, as applied to Atari 2600 game-playing, has emphasized tackling difficult exploration problems such as Montezuma's Revenge (Bellemare et al., 2016). Recently, bonus-based…

Atari Games Montezuma's Revenge Reinforcement Learning