MSD-Kmeans: A Novel Algorithm for Efficient Detection of Global and Local Outliers

2019-10-15 · Yuanyuan Wei, Julian Jang-Jaccard, Fariza Sabrina, Timothy McIntosh

Outlier detection is a technique in data mining that aims to detect unusual or unexpected records in the dataset. Existing outlier detection algorithms have different pros and cons and exhibit different sensitivity to noisy data such as extreme values. In this paper, we propose a novel cluster-based outlier detection algorithm named MSD-Kmeans that combines the statistical method of Mean and Standard Deviation (MSD) and the machine learning clustering algorithm K-means to detect outliers more accurately with the better control of extreme values. There are two phases in this combination method of MSD-Kmeans: (1) applying MSD algorithm to eliminate as many noisy data to minimize the interference on clusters, and (2) applying K-means algorithm to obtain local optimal clusters. We evaluate our algorithm and demonstrate its effectiveness in the context of detecting possible overcharging of taxi fares, as greedy dishonest drivers may attempt to charge high fares by detouring. We compare the performance indicators of MSD-Kmeans with those of other outlier detection algorithms, such as MSD, K-means, Z-score, MIQR and LOF, and prove that the proposed MSD-Kmeans algorithm achieves the highest measure of precision, accuracy, and F-measure. We conclude that MSD-Kmeans can be used for effective and efficient outlier detection on data of varying quality on IoT devices.

📄 PDF Abstract BibTeX arXiv:1910.06588

Code (0)

등록된 구현이 없습니다.

Tasks

ClusteringOutlier Detection

Local Subspace-Based Outlier Detection using Global Neighbourhoods

2016-11-01 · Bas van Stein, Matthijs van Leeuwen, Thomas Bäck

Outlier detection in high-dimensional data is a challenging yet important task, as it has applications in, e.g., fraud detection and quality control. State-of-the-art density-based algorithms perform well because they 1)…

Fraud Detection Outlier Detection

Notes on using Determinantal Point Processes for Clustering with Applications to Text Clustering

2014-10-26 · Apoorv Agarwal, Anna Choromanska, Krzysztof Choromanski

In this paper, we compare three initialization schemes for the KMEANS clustering algorithm: 1) random initialization (KMEANSRAND), 2) KMEANS++, and 3) KMEANSD++. Both KMEANSRAND and KMEANS++ have a major that the value o…

Clustering Point Processes Text Clustering

Semi-supervised Vector-Quantization in Visual SLAM using HGCN

2022-07-14 · Amir Zarringhalam, Saeed Shiry Ghidary, Ali Mohades Khorasani

In this paper, two semi-supervised appearance based loop closure detection technique, HGCN-FABMAP and HGCN-BoW are introduced. Furthermore an extension to the current state of the art localization SLAM algorithm, ORB-SLA…

Decision Making Loop Closure Detection Quantization

ECORS: An Ensembled Clustering Approach to Eradicate The Local And Global Outlier In Collaborative Filtering Recommender System

2024-10-01 · Mahamudul Hasan

Recommender systems are designed to suggest items based on user preferences, helping users navigate the vast amount of information available on the internet. Given the overwhelming content, outlier detection has emerged …

Clustering Collaborative Filtering Navigate Outlier Detection+1

Interpreting Outliers: Localized Logistic Regression for Density Ratio Estimation

2017-02-21 · Makoto Yamada, Song Liu, Samuel Kaski

We propose an inlier-based outlier detection method capable of both identifying the outliers and explaining why they are outliers, by identifying the outlier-specific features. Specifically, we employ an inlier-based out…

Density Ratio Estimation Outlier Detection regression Variable Selection