Reproducibility Requires Consolidated Artifacts
Machine learning is facing a 'reproducibility crisis' where a significant number of works report failures when attempting to reproduce previously published results. We evaluate the sources of reproducibility failures using a meta-analysis of 142 replication studies from ReScience C and 204 code repositories. We find that missing experiment details such as hyperparameters are potential causes of unreproducibility. We experimentally show the bias of different hyperparameter selection strategies and conclude that consolidated artifacts with a unified framework can help support reproducibility.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
CG-SENSE revisited: Results from the first ISMRM reproducibility challenge
Purpose: The aim of this work is to shed light on the issue of reproducibility in MR image reconstruction in the context of a challenge. Participants had to recreate the results of "Advances in sensitivity encoding with …
Image ReconstructionSSIMFrom Data to Decision: Data-Centric Infrastructure for Reproducible ML in Collaborative eScience
Reproducibility remains a central challenge in machine learning (ML), especially in collaborative eScience projects where teams iterate over data, features, and models. Current ML workflows are often dynamic yet fragment…
Referencing Source Code Artifacts: a Separate Concern in Software Citation
Among the entities involved in software citation, software source code requires special attention, due to the role it plays in ensuring scientific reproducibility. To reference source code we need identifiers that are no…
Harnessing Optimization Dynamics for Curvature-Informed Model Merging
Model merging is an effective post-training strategy for composing capabilities in large language models without joint retraining. We study this in the supervised fine-tuning (SFT) stage, where multiple capability-based …
Instruction FollowingAssessing Reproducibility in Evolutionary Computation: A Case Study using Human- and LLM-based Assessment
Reproducibility is an important requirement in evolutionary computation, where results largely depend on computational experiments. In practice, reproducibility relies on how algorithms, experimental protocols, and artif…