Long-term Reproducibility for Neural Architecture Search
It is a sad reflection of modern academia that code is often ignored after publication -- there is no academic 'kudos' for bug fixes / maintenance. Code is often unavailable or, if available, contains bugs, is incomplete, or relies on out-of-date / unavailable libraries. This has a significant impact on reproducibility and general scientific progress. Neural Architecture Search (NAS) is no exception to this, with some prior work in reproducibility. However, we argue that these do not consider long-term reproducibility issues. We therefore propose a checklist for long-term NAS reproducibility. We evaluate our checklist against common NAS approaches along with proposing how we can retrospectively make these approaches more long-term reproducible.
Code (1)
Tasks
Neural Architecture SearchSimilar Papers 제목 키워드 기반
End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF: A Reproducibility Study
We present a reproducibility study of the state-of-the-art neural architecture for sequence labeling proposed by Ma and Hovy (2016)\cite{ma2016end}. The original BiLSTM-CNN-CRF model combines character-level representati…
Structured PredictionReproducible White Matter Tract Segmentation Using 3D U-Net on a Large-scale DTI Dataset
Tract-specific diffusion measures, as derived from brain diffusion MRI, have been linked to white matter tract structural integrity and neurodegeneration. As a consequence, there is a large interest in the automatic segm…
Diffusion MRISegmentationGenerative Adversarial Neural Architecture Search
Despite the empirical success of neural architecture search (NAS) in deep learning applications, the optimality, reproducibility and cost of NAS schemes remain hard to assess. In this paper, we propose Generative Adversa…
Neural Architecture SearchLarge Language Models for Software Engineering: A Reproducibility Crisis
Reproducibility is a cornerstone of scientific progress, yet its state in large language model (LLM)-based software engineering (SE) research remains poorly understood. This paper presents the first large-scale, empirica…
Research Reproducibility as a Survival Analysis
There has been increasing concern within the machine learning community that we are in a reproducibility crisis. As many have begun to work on this problem, all work we are aware of treat the issue of reproducibility as …
Survival Analysis