We Need to Talk About Random Splits
Gorman and Bedrick (2019) argued for using random splits rather than standard splits in NLP experiments. We argue that random splits, like standard splits, lead to overly optimistic performance estimates. We can also split data in biased or adversarial ways, e.g., training on short sentences and evaluating on long ones. Biased sampling has been used in domain adaptation to simulate real-world drift; this is known as the covariate shift assumption. In NLP, however, even worst-case splits, maximizing bias, often under-estimate the error observed on new samples of in-domain data, i.e., the data that models should minimally generalize to at test time. This invalidates the covariate shift assumption. Instead of using multiple random splits, future benchmarks should ideally include multiple, independent test sets instead; if infeasible, we argue that multiple biased splits leads to more realistic performance estimates than multiple random splits.
Code (1)
Tasks
Domain AdaptationSimilar Papers 제목 키워드 기반
We Need to Talk about Standard Splits
It is standard practice in speech {\&} language technology to rank systems according to their performance on a test set held out for evaluation. However, few researchers apply statistical tests to determine whether diffe…
We Need to Talk About train-dev-test Splits
Standard train-dev-test splits used to benchmark multiple models against each other are ubiquitously used in Natural Language Processing (NLP). In this setup, the train data is used for training the model, the developmen…
Model SelectionIt's about Time: Rethinking Evaluation on Rumor Detection Benchmarks using Chronological Splits
New events emerge over time influencing the topics of rumors in social media. Current rumor detection benchmarks use random splits as training, development and test sets which typically results in topical overlaps. Conse…
We need to talk about random seeds
Modern neural network libraries all take as a hyperparameter a random seed, typically used to determine the initial state of the model parameters. This opinion piece argues that there are some safe uses for random seeds:…
We need to talk about random seeds
Modern neural network libraries all take as a hyperparameter a random seed, typically used to determine the initial state of the model parameters. This position piece argues that there are some safe uses for random seeds…
Position