UserBERT: Self-supervised User Representation Learning
This paper extends the BERT model to user data for pretraining user representations in a self-supervised way. By viewing actions (e.g., purchases and clicks) in behavior sequences (i.e., usage history) in an analogous way to words in sentences, we propose methods for the tokenization, the generation of input representation vectors and a novel pretext task to enable the pretraining model to learn from its own input, omitting the burden of collecting additional data. Further, our model adopts a unified structure to simultaneously learn from long-term and short-term user behavior as well as user profiles. Extensive experiments demonstrate that the learned representations result in significant improvements when transferred to three different real-world tasks, particularly in comparison with task-specific modeling and representations obtained from multi-task learning.
Code (0)
등록된 구현이 없습니다.
Tasks
Multi-Task LearningRepresentation LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
UserBERT: Contrastive User Model Pre-training
User modeling is critical for personalized web applications. Existing user modeling methods usually train user models from user behaviors with task-specific labeled data. However, labeled data in a target task may be ins…
modelUserBERT: Modeling Long- and Short-Term User Preferences via Self-Supervision
E-commerce platforms generate vast amounts of customer behavior data, such as clicks and purchases, from millions of unique users every day. However, effectively using this data for behavior understanding tasks is challe…
Representation LearningExploiting Behavioral Consistence for Universal User Representation
User modeling is critical for developing personalized services in industry. A common way for user modeling is to learn user representations that can be distinguished by their interests or preferences. In this work, we fo…
Self-Supervised LearningLearning Universal User Representations via Self-Supervised Lifelong Behaviors Modeling
Universal user representation is an important research topic in industry, and is widely used in diverse downstream user analysis tasks, such as user profiling and user preference prediction. With the rapid development of…
Contrastive LearningDimensionality ReductionRepresentation LearningSelf-Supervised LearningSelfReplay: Adapting Self-Supervised Sensory Models via Adaptive Meta-Task Replay
Self-supervised learning has emerged as a method for utilizing massive unlabeled data for pre-training models, providing an effective feature extractor for various mobile sensing applications. However, when deployed to e…
Computational EfficiencyDomain AdaptationMeta-LearningSelf-Supervised Learning