We Need to Talk About train-dev-test Splits
Standard train-dev-test splits used to benchmark multiple models against each other are ubiquitously used in Natural Language Processing (NLP). In this setup, the train data is used for training the model, the development set for evaluating different versions of the proposed model(s) during development, and the test set to confirm the answers to the main research question(s). However, the introduction of neural networks in NLP has led to a different use of these standard splits; the development set is now often used for model selection during the training procedure. Because of this, comparing multiple versions of the same model during development leads to overestimation on the development data. As an effect, people have started to compare an increasing amount of models on the test data, leading to faster overfitting and “expiration” of our test sets. We propose to use a tune-set when developing neural network methods, which can be used for model picking so that comparing the different versions of a new model can safely be done on the development data.
Code (1)
Tasks
Model SelectionSimilar Papers 제목 키워드 기반
We Need to Talk about Standard Splits
It is standard practice in speech {\&} language technology to rank systems according to their performance on a test set held out for evaluation. However, few researchers apply statistical tests to determine whether diffe…
We Need to Talk About Random Splits
Gorman and Bedrick (2019) argued for using random splits rather than standard splits in NLP experiments. We argue that random splits, like standard splits, lead to overly optimistic performance estimates. We can also spl…
Domain AdaptationWhat Would You Ask the Machine Learning Model? Identification of User Needs for Model Explanations Based on Human-Model Conversations
Recently we see a rising number of methods in the field of eXplainable Artificial Intelligence. To our surprise, their development is driven by model developers rather than a study of needs for human end users. The analy…
ChatbotExplainable artificial intelligencemodelAuditory distraction in open-plan office environments: The effect of multi-talker acoustics
Within the soundscapes of open-plan offices, irrelevant speech has consistently been reported as the most distracting, and causing performance decrements for workers. Notwithstanding this generalization, the 'babble' cre…
Experimental DesignSmall Effect Sizes in Malware Detection? Make Harder Train/Test Splits!
Industry practitioners care about small improvements in malware detection accuracy because their models are deployed to hundreds of millions of machines, meaning a 0.1\% change can cause an overwhelming number of false p…
Malware Detection