The Effect of Third Party Implementations on Reproducibility
Reproducibility of recommender systems research has come under scrutiny during recent years. Along with works focusing on repeating experiments with certain algorithms, the research community has also started discussing various aspects of evaluation and how these affect reproducibility. We add a novel angle to this discussion by examining how unofficial third-party implementations could benefit or hinder reproducibility. Besides giving a general overview, we thoroughly examine six third-party implementations of a popular recommender algorithm and compare them to the official version on five public datasets. In the light of our alarming findings we aim to draw the attention of the research community to this neglected aspect of reproducibility.
Code (5)
Tasks
Recommendation SystemsSimilar Papers 제목 키워드 기반
Stdlib or Third-Party? Empirical Performance and Correctness of LLM-Assisted Zero-Dependency Python Libraries
Third-party Python libraries introduce dependency management overhead, supply chain risk, and deployment friction in constrained environments. A natural question is how much of this ecosystem can be replicated using only…
Online vs Offline: A Comparative Study of First-Party and Third-Party Evaluations of Social Chatbots
This paper explores the efficacy of online versus offline evaluation methods in assessing conversational chatbots, specifically comparing first-party direct interactions with third-party observational assessments. By ext…
BenchmarkingChatbotThird-Party Aligner for Neural Word Alignments
Word alignment is to find translationally equivalent words between source and target sentences. Previous work has demonstrated that self-training can achieve competitive word alignment results. In this paper, we propose …
Language ModelingLanguage ModellingWord AlignmentAttacks on Third-Party APIs of Large Language Models
Large language model (LLM) services have recently begun offering a plugin ecosystem to interact with third-party API services. This innovation enhances the capabilities of LLMs, but it also introduces risks, as these plu…
Language ModelingLanguage ModellingLarge Language ModelThird-Party Credit Guarantees and the Cost of Debt: Evidence from Corporate Loans
Using a comprehensive dataset collected by the Federal Reserve, I find that over one-third of corporate loans issued by US banks are fully guaranteed by legal entities separate from borrowing firms. Using an empirical st…