ClusterDataSplit: Exploring Challenging Clustering-Based Data Splits for Model Performance Evaluation
This paper adds to the ongoing discussion in the natural language processing community on how to choose a good development set. Motivated by the real-life necessity of applying machine learning models to different data distributions, we propose a clustering-based data splitting algorithm. It creates development (or test) sets which are lexically different from the training data while ensuring similar label distributions. Hence, we are able to create challenging cross-validation evaluation setups while abstracting away from performance differences resulting from label distribution shifts between training and test data. In addition, we present a Python-based tool for analyzing and visualizing data split characteristics and model performance. We illustrate the workings and results of our approach using a sentiment analysis and a patent classification task.
Code (1)
Tasks
ClusteringPatent classificationSentiment AnalysisSimilar Papers 제목 키워드 기반
A clustering approach to heterogeneous change detection
Change detection in heterogeneous multitemporal satellite images is a challenging and still not much studied topic in remote sensing and earth observation. This paper focuses on comparison of image pairs covering the sam…
Change DetectionClusteringEarth ObservationTree-Guided $L_1$-Convex Clustering
Convex clustering is a modern clustering framework that guarantees globally optimal solutions and performs comparably to other advanced clustering methods. However, obtaining a complete dendrogram (clusterpath) for large…
ClusteringComputational EfficiencyDistributed ComputingA new type of federated clustering: A non-model-sharing approach
In recent years, the growing need to leverage sensitive data across institutions has led to increased attention on federated learning (FL), a decentralized machine learning paradigm that enables model training without sh…
ClusteringFederated LearningK-Splits: Improved K-Means Clustering Algorithm to Automatically Detect the Number of Clusters
This paper introduces k-splits, an improved hierarchical algorithm based on k-means to cluster data without prior knowledge of the number of clusters. K-splits starts from a small number of clusters and uses the most sig…
ClusteringPositionOne-Shot Coresets: The Case of k-Clustering
Scaling clustering algorithms to massive data sets is a challenging task. Recently, several successful approaches based on data summarization methods, such as coresets and sketches, were proposed. While these techniques …
ClusteringData SummarizationSmall Data Image Classification