Interpretable Single-dimension Outlier Detection (ISOD): An Unsupervised Outlier Detection Method Based on Quantiles and Skewness Coefficients
Source abstract
Abstract
A crucial area of study in data mining is outlier detection, particularly in the areas of network security, credit card fraud detection, industrial flaw detection, etc. Existing outlier detection algorithms, which can be divided into supervised methods, semi-supervised methods, and unsupervised methods, suffer from missing labeled data, the curse of dimensionality, low interpretability, etc. To address these issues, in this paper, we present an unsupervised outlier detection method based on quantiles and skewness coefficients called ISOD (Interpretable Single dimension Outlier Detection). ISOD first fulfils the empirical cumulative distribution function before computing the quantile and skewness coefficients of each dimension. Finally, it outputs the outlier score. This paper’s contributions are as follows: (1) we propose an unsupervised outlier detection algorithm called ISOD, which has high interpretability and scalability; (2) massive experiments on benchmark datasets demonstrated the superior performance of the ISOD algorithm compared with state-of-the-art baselines in terms of ROC and AP.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Anomaly Detection Techniques and Applications · Imbalanced Data Classification Techniques · Network Security and Intrusion Detection
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science and Information Security & School of Software Engineering, Guilin University of Electronic Technology, Guilin 541004, China
Guangxi Key Laboratory of Cryptography and Information Security, Guilin 541004, China
Funding
National Natural Science Foundation of China · Guangxi Natural Science Foundation · Innovation Project of Guangxi Graduate Education
References (48)
Show source-provided references
1. Aggarwal, C.C., and Aggarwal, C.C. (2017). An Introduction to Outlier Analysis, Springer.10.1007/978-3-319-47578-3 ↗
2. Anomaly detection: A survey10.1145/1541880.1541882 ↗
3. A meta-level analysis of online anomaly detectors10.1007/s00778-022-00773-x ↗
4. ICDF: Intrusion collaborative detection framework based on confidence10.1002/int.22877 ↗
5. Heigl, M., Weigelt, E., Urmann, A., Fiala, D., and Schramm, M. (2021). Exploiting the Outcome of Outlier Detection for Novel Attack Pattern Recognition on Streaming Data. Electronics, 10.10.3390/electronics10172160 ↗
6. Automatic Traffic Anomaly Detection on the Road Network with Spatial-Temporal Graph Neural Network Representation Learning
7. Erroneous High Occupancy Vehicle Lane Data: Detecting Misconfigured Traffic Sensors With Machine Learning10.1177/03611981221126515 ↗
8. Anomaly detection in autonomous electric vehicles using AI techniques: A comprehensive survey10.1111/exsy.12754 ↗
9. A Dynamic Deep Reinforcement Learning-Bayesian Framework for Anomaly Detection10.1109/TITS.2022.3200906 ↗
10. Intelligent video anomaly detection and classification using faster RCNN with deep reinforcement learning model10.1016/j.imavis.2021.104229 ↗
11. Zhao, Y., Deng, B., Shen, C., Liu, Y., Lu, H., and Hua, X.-S. (2017, January 23–27). Spatio-Temporal AutoEncoder for Video Anomaly Detection. Proceedings of the 25th ACM International Conference on Multimedia (MM), Comp Hist Museum, Mountain View, CA, USA.10.1145/3123266.3123451 ↗
12. Dang, T.T., Ngan, H.E.T., and Liu, W. (2015, January 21–24). Distance-Based k-Nearest Neighbors Outlier Detection Method in Large-Scale Traffic Data. Proceedings of the IEEE International Conference on Digital Signal Processing (DSP), Singapore.10.1109/ICDSP.2015.7251924 ↗
13. Integrating Machine Learning Algorithms With Quantum Annealing Solvers for Online Fraud Detection10.1109/ACCESS.2022.3190897 ↗
14. KAGO: An approximate adaptive grid-based outlier detection approach using kernel density estimate10.1007/s10044-021-00998-6 ↗
15. Distributed Deep Forest and its Application to Automatic Detection of Cash-Out Fraud
16. Chaudhry, H.N., Javed, Y., Kulsoom, F., Mehmood, Z., Khan, Z.I., Shoaib, U., and Janjua, S.H. (2021). Sentiment Analysis of before and after Elections: Twitter Data of U.S. Election 2020. Electronics, 10.10.3390/electronics10172082 ↗
17. Chalapathy, R., Toth, E., and Chawla, S. (2020, January 10–14). Group Anomaly Detection Using Deep Generative Models. Proceedings of the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD), Dublin, Ireland.10.1007/978-3-030-10925-7_11 ↗
18. Chenaghlou, M., Moshtaghi, M., Leckie, C., and Salehi, M. (2018, January 3–6). Online Clustering for Evolving Data Streams with Online Anomaly Detection. Proceedings of the 22nd Pacific-Asia Conference on Knowledge Discovery and Data Mining (PAKDD), Melbourne, Australia.10.1007/978-3-319-93037-4_40 ↗
19. NHAD: Neuro-Fuzzy Based Horizontal Anomaly Detection in Online Social Networks
20. A survey of outlier detection in high dimensional data streams10.1016/j.cosrev.2022.100463 ↗
21. Pei, Y., and Zaïane, O. (2023, October 20). A Synthetic Data Generator for Clustering and Outlier Analysis. Available online: https://era.library.ualberta.ca/items/63beb6a7-cc50-4ffd-990b-64723b1e4bf9.
22. Sikder, M.N.K., and Batarseh, F.A. (2023). AI Assurance, Elsevier.
23. IoT anomaly detection methods and applications: A survey10.1016/j.iot.2022.100568 ↗
24. Deep Learning for Anomaly Detection10.1145/3439950 ↗
25. Outlier Detection: Methods, Models, and Classification
26. Samudra, S., Barbosh, M., and Sadhu, A. (2023). Machine Learning-Assisted Improved Anomaly Detection for Structural Health Monitoring. Sensors, 23.10.3390/s23073365 ↗
27. Qiu, J., Shi, H., Hu, Y., and Yu, Z. (2023). Enhancing Anomaly Detection Models for Industrial Applications through SVM-Based False Positive Classification. Appl. Sci., 13.10.3390/app132312655 ↗
28. Kerboua, A., and Kelaiaia, R. (2023). Fault Diagnosis in an Asynchronous Motor Using Three-Dimensional Convolutional Neural Network. Arab. J. Sci. Eng., 1–19.10.1007/s13369-023-08025-y ↗
29. Masked swin transformer unet for industrial anomaly detection10.1109/TII.2022.3199228 ↗
30. Drost, B., Ulrich, M., Bergmann, P., Hartinger, P., and Steger, C. (2017, January 22–29). Introducing mvtec itodd-a dataset for 3d object recognition in industry. Proceedings of the IEEE International Conference on Computer Vision Workshops, Venice, Italy.10.1109/ICCVW.2017.257 ↗
31. A Comparative Study for Outlier Detection Methods in High Dimensional Text Data10.2478/jaiscr-2023-0001 ↗
32. Sunny, J.S., Patro, C.P.K., Karnani, K., Pingle, S.C., Lin, F., Anekoji, M., Jones, L.D., Kesari, S., and Ashili, S. (2022). Anomaly Detection Framework for Wearables Data: A Perspective Review on Data Concepts, Data Analysis Algorithms and Prospects. Sensors, 22.10.3390/s22030756 ↗
33. Liu, F.T., Ting, K.M., and Zhou, Z.-H. (2008, January 15–19). Isolation Forest. Proceedings of the 8th IEEE International Conference on Data Mining, Pisa, Italy.10.1109/ICDM.2008.17 ↗
34. Staffini, A., Svensson, T., Chung, U.-I., and Svensson, A.K. (2023). A Disentangled VAE-BiLSTM Model for Heart Rate Anomaly Detection. Bioengineering, 10.10.3390/bioengineering10060683 ↗
35. Generic and scalable periodicity adaptation framework for time-series anomaly detection10.1007/s11042-022-13304-1 ↗
36. Huang, Y., Liu, W., Li, S., Guo, Y., and Chen, W. (2023). A Novel Unsupervised Outlier Detection Algorithm Based on Mutual Information and Reduced Spectral Clustering. Electronics, 12.10.3390/electronics12234864 ↗
37. Mozaffari, M., Doshi, K., and Yilmaz, Y. (2023). Self-Supervised Learning for Online Anomaly Detection in High-Dimensional Data Streams. Electronics, 12.10.3390/electronics12091971 ↗
38. Liu, Y., Zhou, S., Wan, Z., Qiu, Z., Zhao, L., Pang, K., Li, C., and Yin, Z. (2023). A Self-Supervised Anomaly Detector of Fruits Based on Hyperspectral Imaging. Foods, 12.10.3390/foods12142669 ↗
39. Deep anomaly detection with self-supervised learning and adversarial training10.1016/j.patcog.2021.108234 ↗
40. Hojjati, H., Ho, T.K.K., and Armanfard, N. (2022). Self-Supervised Anomaly Detection: A Survey and Outlook. arXiv.
41. Liu, K., Fu, Y., Wang, P., Wu, L., Bo, R., and Li, X. (2019, January 4–8). Automating feature subspace exploration via multi-agent reinforcement learning. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, Anchorage, AK, USA.10.1145/3292500.3330868 ↗
42. Ramaswamy, S., Rastogi, R., and Shim, K. (2000, January 16–18). Efficient algorithms for mining outliers from large data sets. Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data, Dallas, TX, USA.10.1145/342009.335437 ↗
43. LOF: Identifying density-based local outliers10.1145/335191.335388 ↗
44. Discovering cluster-based local outliers10.1016/S0167-8655(03)00003-5 ↗
45. Zhao, Y., Nasrullah, Z., Hryniewicki, M.K., and Li, Z. (2019, January 2–4). LSCP: Locally selective combination in parallel outlier ensembles. Proceedings of the 2019 SIAM International Conference on Data Mining, Calgary, AB, Canada.10.1137/1.9781611975673.66 ↗
46. Support vector method for novelty detection
47. Deep isolation forest for anomaly detection10.1109/TKDE.2023.3270293 ↗
48. Akcay, S., Atapour-Abarghouei, A., and Breckon, T.P. (2018, January 2–6). GANomaly: Semi-supervised Anomaly Detection via Adversarial Training. Proceedings of the 14th Asian Conference on Computer Vision (ACCV), Perth, Australia.10.1007/978-3-030-20893-6_39 ↗