Does It Look Sequential? An Analysis of Datasets for Evaluation of Sequential Recommendations
Sequential recommender systems are an important and demanded area of research. Such systems aim to use the order of interactions in a user's history to predict future interactions. The premise is that the order of interactions and sequential patterns play an essential role. Therefore, it is crucial to use datasets that exhibit a sequential structure to evaluate sequential recommenders properly. We apply several methods based on the random shuffling of the user's sequence of interactions to assess the strength of sequential structure across 15 datasets, frequently used for sequential recommender systems evaluation in recent research papers presented at top-tier conferences. As shuffling explicitly breaks sequential dependencies inherent in datasets, we estimate the strength of sequential patterns by comparing metrics for shuffled and original versions of the dataset. Our findings show that several popular datasets have a rather weak sequential structure.
Code (1)
Tasks
Recommendation SystemsSimilar Papers 제목 키워드 기반
Disentanglement Beyond Static vs. Dynamic: A Benchmark and Evaluation Framework for Multi-Factor Sequential Representations
Learning disentangled representations in sequential data is a key goal in deep learning, with broad applications in vision, audio, and time series. While real-world data involves multiple interacting semantic factors ove…
How Well Does Self-Supervised Pre-Training Perform with Streaming Data?
Prior works on self-supervised pre-training focus on the joint training scenario, where massive unlabeled data are assumed to be given as input all at once, and only then is a learner trained. Unfortunately, such a probl…
Representation LearningSelf-Supervised LearningHow Well Does Self-Supervised Pre-Training Perform with Streaming ImageNet?
Prior works on self-supervised pre-training focus on the joint training scenario, where massive unlabeled data are assumed to be given as input all at once, and only then is a learner trained. Unfortunately, such a probl…
Self-Supervised LearningQuality Does Matter: A Detailed Look at the Quality and Utility of Web-Mined Parallel Corpora
We conducted a detailed analysis on the quality of web-mined corpora for two low-resource languages (making three language pairs, English-Sinhala, English-Tamil and Sinhala-Tamil). We ranked each corpus according to a si…
Machine TranslationNMTTranslationInterpretable Multi-dataset Evaluation for Named Entity Recognition
With the proliferation of models for natural language processing tasks, it is even harder to understand the differences between models and their relative merits. Simply looking at differences between holistic metrics suc…
named-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)NER