Community Perspective on Replicability in Natural Language Processing
With recent efforts in drawing attention to the task of replicating and/or reproducing results, for example in the context of COLING 2018 and various LREC workshops, the question arises how the NLP community views the topic of replicability in general. Using a survey, in which we involve members of the NLP community, we investigate how our community perceives this topic, its relevance and options for improvement. Based on over two hundred participants, the survey results confirm earlier observations, that successful reproducibility requires more than having access to code and data. Additionally, the results show that the topic has to be tackled from the authors{'}, reviewers{'} and community{'}s side.
Code (0)
등록된 구현이 없습니다.
Tasks
SurveySimilar Papers 제목 키워드 기반
Replicability of Research in Biomedical Natural Language Processing: a pilot evaluation for a coding task
Replicability Analysis for Natural Language Processing: Testing Significance with Multiple Datasets
With the ever-growing amounts of textual data from a large variety of languages, domains, and genres, it has become standard to evaluate NLP algorithms on multiple datasets in order to ensure consistent performance acros…
Dependency ParsingGeneral ClassificationPOSPOS Tagging+3Word vectors, reuse, and replicability: Towards a community repository of large-text resources
Defining Replicability of Prediction Rules
In this article I propose an approach for defining replicability for prediction rules. Motivated by a recent NAS report, I start from the perspective that replicability is obtaining consistent results across studies suit…
PredictionFrom Model Performance to Claim: How a Change of Focus in Machine Learning Replicability Can Help Bridge the Responsibility Gap
Two goals - improving replicability and accountability of Machine Learning research respectively, have accrued much attention from the AI ethics and the Machine Learning community. Despite sharing the measures of improvi…
Ethics