Distributed ReliefF based Feature Selection in Spark
Feature selection (FS) is a key research area in the machine learning and data mining fields, removing irrelevant and redundant features usually helps to reduce the effort required to process a dataset while maintaining or even improving the processing algorithm's accuracy. However, traditional algorithms designed for executing on a single machine lack scalability to deal with the increasing amount of data that has become available in the current Big Data era. ReliefF is one of the most important algorithms successfully implemented in many FS applications. In this paper, we present a completely redesigned distributed version of the popular ReliefF algorithm based on the novel Spark cluster computing model that we have called DiReliefF. Spark is increasing its popularity due to its much faster processing times compared with Hadoop's MapReduce model implementation. The effectiveness of our proposal is tested on four publicly available datasets, all of them with a large number of instances and two of them with also a large number of features. Subsets of these datasets were also used to compare the results to a non-distributed implementation of the algorithm. The results show that the non-distributed implementation is unable to handle such large volumes of data without specialized hardware, while our design can process them in a scalable way with much better processing times and memory usage.
Code (0)
등록된 구현이 없습니다.
Tasks
feature selectionSimilar Papers 제목 키워드 기반
An Information Theoretic Feature Selection Framework for Big Data under Apache Spark
With the advent of extremely high dimensional datasets, dimensionality reduction techniques are becoming mandatory. Among many techniques, feature selection has been growing in interest as an important tool to identify r…
Dimensionality Reductionfeature selectionDistributed Correlation-Based Feature Selection in Spark
CFS (Correlation-Based Feature Selection) is an FS algorithm that has been successfully applied to classification problems in many domains. We describe Distributed CFS (DiCFS) as a completely redesigned, scalable, parall…
feature selectionAn Information Theory-Based Feature Selection Framework for Big Data Under Apache Spark
With the advent of extremely high dimensional datasets, dimensionality reduction techniques are becoming mandatory. Of the many techniques available, feature selection (FS) is of growing interest for its ability to ident…
Dimensionality Reductionfeature selectionFeature selection in high-dimensional dataset using MapReduce
This paper describes a distributed MapReduce implementation of the minimum Redundancy Maximum Relevance algorithm, a popular feature selection method in bioinformatics and network inference problems. The proposed approac…
feature selectionVocal Bursts Intensity PredictionBrain tumour classification using BoF-SURF with filter-based feature selection methods
Currently, cancer is a global concern with a focus on reducing its incidence and advancing diagnostic techniques. Faster and more precise cancer cell detection improves treatment and survival prospects. The objective of …
Cell DetectionDiagnosticfeature selectionTumour Classification