The Effect of Data Partitioning Strategy on Model Generalizability: A Case Study of Morphological Segmentation
Recent work to enhance data partitioning strategies for more realistic model evaluation face challenges in providing a clear optimal choice. This study addresses these challenges, focusing on morphological segmentation and synthesizing limitations related to language diversity, adoption of multiple datasets and splits, and detailed model comparisons. Our study leverages data from 19 languages, including ten indigenous or endangered languages across 10 language families with diverse morphological systems (polysynthetic, fusional, and agglutinative) and different degrees of data availability. We conduct large-scale experimentation with varying sized combinations of training and evaluation sets as well as new test data. Our results show that, when faced with new test data: (1) models trained from random splits are able to achieve higher numerical scores; (2) model rankings derived from random splits tend to generalize more consistently.
Code (0)
등록된 구현이 없습니다.
Tasks
DiversitySimilar Papers 제목 키워드 기반
CohortFinder: an open-source tool for data-driven partitioning of biomedical image cohorts to yield robust machine learning models
Batch effects (BEs) refer to systematic technical differences in data collection unrelated to biological variations whose noise is shown to negatively impact machine learning (ML) model generalizability. Here we release …
Uniform Hypergraph Partitioning: Provable Tensor Methods and Sampling Techniques
In a series of recent works, we have generalised the consistency results in the stochastic block model literature to the case of uniform and non-uniform hypergraphs. The present paper continues the same line of study, wh…
Clusteringhypergraph partitioningStochastic Block ModelA Super-pixel-based Approach to the Stable Interpretation of Neural Networks
Saliency maps are widely used in the computer vision community for interpreting neural network classifiers. However, due to the randomness of training samples and optimization algorithms, the resulting saliency maps suff…
P3-Distributed Deep Graph Learning at Scale
Graph Neural Networks (GNNs) have gained significant attention in the recent past, and become one of the fastest growing subareas in deep learning. While several new GNN architectures have been proposed, the scale of rea…
Graph LearningGeneralizable Resource Allocation in Stream Processing via Deep Reinforcement Learning
This paper considers the problem of resource allocation in stream processing, where continuous data flows must be processed in real time in a large distributed system. To maximize system throughput, the resource allocati…
DecoderDeep Reinforcement LearningGraph Embeddinggraph partitioning+3