A Comprehensive Survey for Deep-Learning-Based Abnormality Detection in Smart Grids with Multimodal Image Data
Source abstract
Abstract
In this paper, we provide a comprehensive survey of the recent advances in abnormality detection in smart grids using multimodal image data, which include visible light, infrared, and optical satellite images. The applications in visible light and infrared images, enabling abnormality detection at short range, further include several typical applications in intelligent sensors deployed in smart grids, while optical satellite image data focus on abnormality detection from a large distance. Moreover, the literature in each aspect is organized according to the considered techniques. In addition, several key methodologies and conditions for applying these techniques to abnormality detection are identified to help determine whether to use deep learning and which kind of learning techniques to use. Traditional approaches are also summarized together with their performance comparison with deep-learning-based approaches, based on which the necessity, seen in the surveyed literature, of adopting image-data-based abnormality detection is clarified. Overall, this comprehensive survey categorizes and carefully summarizes insights from representative papers in this field, which will widely benefit practitioners and academic researchers.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Anomaly Detection Techniques and Applications · Remote-Sensing Image Classification · Infrastructure Maintenance and Monitoring
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Electric Power Research Institute, Yunnan Power Grid Co., Ltd., Kunming 650217, China
Shanghai Key Laboratory of Navigation and Location-Based Services, School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, Shanghai 200240, China
Funding
the project · N.S.F.C
References (100+)
Show source-provided references
1. Power system dynamic state estimation: Motivations, definitions, methodologies, and future work10.1109/TPWRS.2019.2894769 ↗
2. Advancing China? s smart grid: Phasor measurement units in a wide-area management system10.1109/MPE.2015.2432372 ↗
3. Anomaly detection: A survey10.1145/1541880.1541882 ↗
4. A systematic method to model power systems as stochastic differential algebraic equations10.1109/TPWRS.2013.2266441 ↗
5. Bollen, M.H. (2000). Understanding power quality problems. Voltage Sags and Interruptions, IEEE Press.10.1109/9780470546840 ↗
6. Dimensionality reduction of synchrophasor data for early event detection: Linearized analysis10.1109/TPWRS.2014.2316476 ↗
7. Invisible units detection and estimation based on random matrix theory10.1109/TPWRS.2019.2935739 ↗
8. Pinto, N., Cruz, D., Monteiro, J., Cabrita, C., Semião, J., Cardoso, P.J., Oliveira, L.M., and Rodrigues, J.M. (2022). IoE-Based Control and Monitoring of Electrical Grids: A Smart Grid’s Perspective. Research Anthology on Smart Grid and Microgrid Development, IGI Global.10.4018/978-1-6684-3666-0.ch037 ↗
9. Moeller, M.S. (2006, January 6–10). Monitoring powerline corridors with stereo satellite imagery. Proceedings of the MAPPS/ASPRS Conference, San Antonio, TX, USA.
10. Shepard, D.P., Bhatti, J.A., Humphreys, T.E., and Fansler, A.A. (2012, January 17–21). Evaluation of smart grid and civilian UAV vulnerability to GPS spoofing attacks. Proceedings of the 25th International Technical Meeting of the Satellite Division of the Institute of Navigation (ION GNSS 2012), Nashville, TN, USA.
11. Flight-testing of a cooperative UGV-to-UAV strategy for improved positioning in challenging GNSS environments10.1016/j.ast.2018.09.035 ↗
12. Automated Power Lines Vegetation Monitoring using High-Resolution Satellite Imagery10.1109/TPWRD.2021.3059307 ↗
13. Li, T., Chen, L., Liu, Y., Liu, J., and Gao, K. (2019, January 26–29). Ultra high voltage power tower SAR interferometry and icing tower testing results. Proceedings of the 2019 6th Asia-Pacific Conference on Synthetic Aperture Radar (APSAR), Xiamen, China.10.1109/APSAR46974.2019.9048254 ↗
14. Ground Displacement Variation around Power Line Corridors on the Loess Plateau Estimated by Persistent Scatterer Interferometry10.1109/ACCESS.2021.3090201 ↗
15. Tarighat, F., Foroughnia, F., and Perissin, D. (2021). Monitoring of power towers’ movement using persistent scatterer SAR interferometry in south west of Tehran. Remote Sens., 13.10.3390/rs13030407 ↗
16. Schwarz, G., Soccorsi, M., Chaabouni-Chouayakh, H., Espinoza, D., Cerra, D., Rodriguez, F., and Datcu, M. (2009, January 12–17). Automated information extraction from high resolution SAR images: TerraSAR-X interpretation applications. Proceedings of the 2009 IEEE International Geoscience and Remote Sensing Symposium, Cape Town, South Africa.10.1109/IGARSS.2009.5417467 ↗
17. Sha, L., Tao, L., Mingzhou, W., Ailing, H., Wenhao, W., Kan, X., and Yan, L. (2014, January 11–14). Study on extra-high voltage power line scatterers in time series SAR. Proceedings of the 2014 Third International Workshop on Earth Observation and Remote Sensing Applications (EORSA), Changsha, China.
18. Power transmission tower monitoring technology based on TerraSAR-X products10.1117/12.912336 ↗
19. Mataloto, B., Ferreira, J.C., and Cruz, N. (2019). LoBEMS—IoT for Building and Energy Management Systems. Electronics, 8.10.3390/electronics8070763 ↗
20. Smart meters for industrial energy conservation and efficiency optimization in Pakistan: Scope, technology and applications10.1016/j.rser.2015.01.004 ↗
21. BRICKS: Building’s reasoning for intelligent control knowledge-based system10.1016/j.scs.2019.101832 ↗
22. Grid of the future10.1109/MPE.2008.931384 ↗
23. Infra-red and ultraviolet control, LVI-testing, partial discharges and another diagnostic methods for detection of electrical equipment’s faults, defects
24. Alam, M.K., Khan, F.H., Johnson, J., and Flicker, J. (2013, January 23–26). PV faults: Overview, modeling, prevention and detection techniques. Proceedings of the IEEE 14th Workshop on Control and Modeling for Power Electronics, Salt Lake City, UT, USA.10.1109/COMPEL.2013.6626400 ↗
25. Advanced Fault Diagnosis for Lithium-Ion Battery Systems: A Review of Fault Mechanisms, Fault Features, and Diagnosis Procedures10.1109/MIE.2020.2964814 ↗
26. Smv, S., Jeg, O., and Fja, C. (2014, January 1–4). Remote Monitoring of High-Voltage Disconnect Switches in Electrical Distribution Substations. Proceedings of the 2014 IEEE 23rd International Symposium on Industrial Electronics (ISIE), Istanbul, Turkey.
27. Novák, M., Biňas, M., and Jakab, F. (2012, January 21–22). Unobtrusive anomaly detection in presence of elderly in a smart-home environment. Proceedings of the 2012 ELEKTRO, Rajecke Teplice, Slovakia.10.1109/ELEKTRO.2012.6225617 ↗
28. Employing entropy measures to identify visitors in multi-occupancy environments10.1007/s12652-020-02824-z ↗
29. Richter, J., Wiede, C., Kaden, S., Weigert, M., and Hirtz, G. (March, January 27). Skin Temperature Measurement based on Human Skeleton Extraction and Infra-red Thermography—An Application of Sensor Fusion Methods in the Field of Physical Training. Proceedings of the 12th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, Porto, Portugal.
30. Sonntag, D., Zillner, S., Smagt, P.V.D., and Lörincz, A. (2017). Overview of the CPS for Smart Factories Project: Deep Learning, Knowledge Acquisition, Anomaly Detection and Intelligent User Interfaces. Industrial Internet of Things, Springer.10.1007/978-3-319-42559-7_19 ↗
31. Ibrahim, M., Alsheikh, A., Awaysheh, F.M., and Alshehri, M.D. (2022). Machine Learning Schemes for Anomaly Detection in Solar Power Plants. Energies, 15.10.3390/en15031082 ↗
32. Barman, B.K., Yadav, S.N., Kumar, S., and Gope, S. (2018, January 1–2). IOT Based Smart Energy Meter for Efficient Energy Utilization in Smart Grid. Proceedings of the 2018 2nd International Conference on Power, Energy and Environment: Towards Smart Technology, Shillong, India.10.1109/EPETSG.2018.8658501 ↗
33. Reviewing the opportunities, challenges, and future directions for the digitalization of energy10.1016/j.erss.2021.102243 ↗
34. Noorwali, A. (2016). Modelling and Analysis of Smart Grids for Critical Data Communication. [Ph.D. Thesis, The University of Western Ontario].
35. Sánchez, V.G., Pfeiffer, C.F., and Skeie, N.O. (2017). A Review of Smart House Analysis Methods for Assisting Older People Living Alone. J. Sens. Actuator Netw., 6.10.3390/jsan6030011 ↗
36. Krishnamurthi, R., Kumar, A., Gopinathan, D., Nayyar, A., and Qureshi, B. (2020). An Overview of IoT Sensor Data Processing, Fusion, and Analysis Techniques. Sensors, 20.10.3390/s20216076 ↗
37. Sensing, communication and security planes: A new challenge for a smart city system design10.1016/j.comnet.2018.08.001 ↗
38. Design and Development of Wired Building Automation Systems10.1016/j.enbuild.2015.02.054 ↗
39. A combined analysis to extract objects in remote sensing images10.1016/S0167-8655(99)00126-9 ↗
40. SVD-based voltage stability assessment from phasor measurement unit data10.1109/TPWRS.2015.2487996 ↗
41. A novel data-driven situation awareness approach for future grids—Using large random matrices for big data modeling10.1109/ACCESS.2018.2805815 ↗
42. Early anomaly detection and localisation in distribution network: A data-driven approach10.1049/iet-gtd.2019.1790 ↗
43. A new approach of exploiting self-adjoint matrix polynomials of large random matrices for anomaly detection and fault location10.1109/TBDATA.2019.2920350 ↗
44. Improving power system state estimation based on matrix-level cleaning10.1109/TPWRS.2020.2984926 ↗
45. A deep learning approach for fault type identification of transmission line
46. Zhao, Z., Xu, G., Qi, Y., Liu, N., and Zhang, T. (2016, January 24–29). Multi-patch deep features for power line insulator status classification from aerial images. Proceedings of the 2016 International Joint Conference on Neural Networks (IJCNN), Vancouver, BC, Canada.10.1109/IJCNN.2016.7727606 ↗
47. Automatic autonomous vision-based power line inspection: A review of current status and the potential role of deep learning10.1016/j.ijepes.2017.12.016 ↗
48. Wide-area monitoring, protection, and control of future electric power networks10.1109/JPROC.2010.2060450 ↗
49. Designing for situation awareness of future power grids: An indicator system based on linear eigenvalue statistics of large random matrices10.1109/ACCESS.2016.2581838 ↗
50. Massive streaming PMU data modelling and analytics in smart grid state evaluation based on multiple high-dimensional covariance test10.1109/TBDATA.2017.2715165 ↗
51. Electric signature detection and analysis for power equipment failure monitoring in smart grid10.1109/TII.2020.3017080 ↗
52. Qiu, R., Sun, H., Hatziargyriou, N., and Poor, H.V. (2016). Big data analysis of power grid from random matrix theory. Chapter 7 of Smarter Energy: From Smart Metering to the Smart Grid, The Institution of Engineering and Technology.
53. Tajer, A., Perlaza, S.M., and Poor, H.V. (2021). Random Matrix Theory for Analyzing Spatio-Temporal Data. Advanced Data Analytics for Power Systems, Cambridge University Press.10.1017/9781108859806 ↗
54. Qiu, R., Chu, L., He, X., Ling, Z., and Liu, H. (2018). Spatiotemporal Big Data Analysis for Smart Grids Based on Random Matrix Theory. Transportation and Power Grid in Smart Cities: Communication Networks and Services, John Wiley & Sons, Ltd.10.1002/9781119360124.ch23 ↗
55. Tajer, A., Perlaza, S.M., and Poor, H.V. (2021). Advanced Data Analytics for Power Systems, Cambridge University Press.10.1017/9781108859806 ↗
56. Survey on anomaly detection using data mining techniques10.1016/j.procs.2015.08.220 ↗
57. Anomaly detection for IoT time-series data: A survey10.1109/JIOT.2019.2958185 ↗
58. Chalapathy, R., and Chawla, S. (2019). Deep learning for anomaly detection: A survey. arXiv.10.1145/3394486.3406704 ↗
59. A comprehensive survey on network anomaly detection10.1007/s11235-018-0475-8 ↗
60. A survey of network anomaly detection techniques10.1016/j.jnca.2015.11.016 ↗
61. Ling, Z., Qiu, R.C., Jin, Z., Zhang, Y., He, X., Liu, H., and Chu, L. (2018). An accurate and real-time self-blast glass insulator location method based on faster R-CNN and U-net with aerial images. arXiv.
62. A texture segmentation algorithm based on PCA and global minimization active contour model for aerial insulator images10.1109/JSTARS.2012.2197672 ↗
63. Cheng, H., Zhai, Y., Chen, R., Wang, D., Dong, Z., and Wang, Y. (2019). Self-shattering defect detection of glass insulators based on spatial features. Energies, 12.10.3390/en12030543 ↗
64. Automatic identification and location technology of glass insulator self-shattering10.1117/1.JEI.26.6.063014 ↗
65. Condition monitoring of 11 kV distribution system insulators incorporating complex imagery using combined DOST-SVM approach10.1109/TDEI.2013.6508770 ↗
66. Insulator condition analysis for overhead distribution lines using combined wavelet support vector machine (SVM)10.1109/TDEI.2010.5412006 ↗
67. Digital image processing approach using combined wavelet hidden Markov model for well-being analysis of insulators10.1049/iet-ipr.2009.0293 ↗
68. Ronneberger, O., Fischer, P., and Brox, T. (2015). U-net: Convolutional networks for biomedical image segmentation. International Conference on Medical image Computing and Computer-Assisted Intervention, Springer.10.1007/978-3-319-24574-4_28 ↗
69. Insulator identification and self-shattering detection based on mask region with convolutional neural network10.1117/1.JEI.28.5.053011 ↗
70. Detection of power line insulator defects using aerial images analyzed with convolutional neural networks10.1109/TSMC.2018.2871750 ↗
71. Zhai, Y., Wang, G., Yu, H., and Wei, G. (2017, January 6–9). Research on the application of the edge detection method for the UAVs icing monitoring of transmission lines. Proceedings of the 2017 IEEE International Conference on Mechatronics and Automation (ICMA), Takamatsu, Japan.10.1109/ICMA.2017.8015972 ↗
72. Zhong, Y.P., Zuo, Q., Zhou, Y., and Zhang, C. (2013, January 15–19). A new image-based algorithm for icing detection and icing thickness estimation for transmission lines. Proceedings of the 2013 IEEE International Conference on Multimedia and Expo Workshops (ICMEW), San Jose, CA, USA.
73. Huang, H., Ma, X., Zhao, L., Du, H., Luo, H., Mao, X., Tang, M., and Liu, Y. (November, January 30). Transmission line icing measurement on photogrammetry method. Proceedings of the MIPPR 2015: Remote Sensing Image Processing, Geographic Information Systems, and Other Applications, Enshi, China.10.1117/12.2203579 ↗
74. A novel method for vegetation encroachment monitoring of transmission lines using a single 2D camera10.1007/s10044-014-0391-9 ↗
75. Intelligent Detection of Vegetation Encroachment of Power Lines with Advanced Stereovision10.1109/TPWRD.2020.3043433 ↗
76. Jain, A., Shah, T., Yousefhussien, M., and Pandey, A. (2021, January 20–25). Combining Remotely Sensed Imagery with Survival Models for Outage Risk Estimation of the Power Grid. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPRW53098.2021.00131 ↗
77. A robust insulator detection algorithm based on local features and spatial orders for aerial images10.1109/LGRS.2014.2369525 ↗
78. Imagenet classification with deep convolutional neural networks
79. Road extraction by deep residual u-net10.1109/LGRS.2018.2802944 ↗
80. McKinnon, C., Carroll, J., McDonald, A., Koukoura, S., Infield, D., and Soraghan, C. (2020). Comparison of new anomaly detection technique for wind turbine condition monitoring using gearbox SCADA data. Energies, 13.10.3390/en13195152 ↗
81. Recherche, D.G. (2006). Vision and Strategy for Europe’s Electricity Networks of the Future: European Technology PlatformSmartGrids, Office for Official Publications of the European Communities.
82. Condition parameter modeling for anomaly detection in wind turbines10.3390/en7053104 ↗
83. Alves, J.M., Carreira, P., Ferreira, H., Pereira, H., and Cardoso, F. (2013, January 10–13). Smart Grids: Asset management in primary distribution substations, with smart sensor networks. Proceedings of the 22nd International Conference and Exhibition on Electricity Distribution (CIRED 2013), Stockholm, Sweden.
84. Moghe, R., Iyer, A., Lambert, F.C., and Divan, D. (2012, January 22–26). A robust smart sensor for smart substations. Proceedings of the 2012 IEEE Power and Energy Society General Meeting, San Diego, CA, USA.10.1109/PESGM.2012.6345149 ↗
85. Yerra, R., Bharathi, A.K., Rajalakshmi, P., and Desai, U.B. (2011, January 6–9). WSN based power monitoring in smart grids. Proceedings of the 2011 Seventh International Conference on Intelligent Sensors, Sensor Networks and Information Processing, Adelaide, Australia.10.1109/ISSNIP.2011.6146589 ↗
86. A Feasibility Study on Autonomous Online Condition Monitoring of High-Voltage Overhead Power Lines10.1109/TIM.2009.2012943 ↗
87. A Miniature Energy Harvesting Device for Wireless Sensors in Electric Power System10.1109/JSEN.2010.2040173 ↗
88. Harvesting energy from magnetic fields to power condition monitoring sensors10.1109/JSEN.2013.2251625 ↗
89. Research Status of Deformation and Inclination Detection of Transmission Towers10.1016/j.powtec.2018.04.011 ↗
90. Design of High precision Acquisition System for Power Grid Data Based on Beidou J
91. Jin, Y., Wang, W., Pei, L., Chen, X., and Song, B. (2021). The Research and Implementation Transmission Line Tower Rod and Monitoring System Using Reverse Network RTK Technology, IOP Publishing. Journal of Physics: Conference Series.10.1088/1742-6596/1894/1/012044 ↗
92. Shen, Z., Yu, W., Song, B., Wang, H., Li, J., Jin, J., Zhang, Y., and Wang, M. (2020, January 20–22). Research on the Tower Pole Tilt Mechanism and Risk Assessment Technology Based on BeiDou reverse network RTK. Proceedings of the E3S Web of Conferences, Changchun, China.10.1051/e3sconf/202016503011 ↗
93. The utilization of satellite images to identify trees endangering transmission lines10.1109/TPWRD.2009.2022664 ↗
94. Ahmad, J., Malik, A.S., and Xia, L. (2011, January 19–20). Vegetation monitoring for high-voltage transmission line corridors using satellite stereo images. Proceedings of the 2011 National Postgraduate Conference, Perak, Malaysia.10.1109/NatPC.2011.6136337 ↗
95. Xiao, C., Qin, R., Huang, X., and Li, J. (2018, January 23–27). Individual tree detection from multi-view satellite images. Proceedings of the IGARSS 2018—2018 IEEE International Geoscience and Remote Sensing Symposium, Valencia, Spain.10.1109/IGARSS.2018.8518040 ↗
96. Henderson, F.M., and Lewis, A.J. (1998). Principles and Applications of Imaging Radar. Manual of Remote Sensing: Volume 2, OSTI.
97. Zhao, Y.H., and Liu, P. (2018). Adaptive Ship Detection for Single-Look Complex SAR Images Based on SVWIE-Noncircularity Decomposition. Sensors, 18.10.3390/s18103293 ↗
98. Principles of GNSS, inertial, and multisensor integrated navigation systems, [Book review]10.1109/MAES.2014.14110 ↗
99. P3-LOAM: PPP/LiDAR Loosely Coupled SLAM With Accurate Covariance Estimation and Robust RAIM in Urban Canyon Environment10.1109/JSEN.2020.3042968 ↗
100. Fast and positive definite estimation of large covariance matrix for high-dimensional data analysis10.1109/TBDATA.2019.2937785 ↗