Sequential Data Processing for IMERG Satellite Rainfall Comparison and Improvement Using LSTM and ADAM Optimizer
Source abstract
Abstract
This study introduces a systematic methodology whereby different technologies were utilized to download, pre-process, and interactively compare the rainfall datasets from the Integrated Multi-Satellite Retrievals for Global Precipitation Mission (IMERG) satellite and rain gauges. To efficiently handle the large volume of data, we developed automated shell scripts for downloading IMERG data and storing it, along with rain gauge data, in a relational database system. Hypertext pre-processor (pHp) programs were built to visualize the result for better analysis. In this study, the performance of IMERG estimations over the east coast of Peninsular Malaysia for the duration of 10 years (2011–2020) against rain gauge observation data is evaluated. Moreover, this study aimed to improve the daily IMERG estimations with long short-term memory (LSTM) developed with Python. Findings show that the LSTM with Adaptive Moment Estimation (ADAM) optimizer trained against the mean square error (MSE) loss enhances the accuracy of satellite estimations. At the point-to-pixel scale, the correlation between satellite estimations and ground observations was increased by about 15%. The bias was reduced by 81–118%, MAE was reduced by 18–59%, the root-mean-square error (RMSE) was reduced by 1–66%, and the Kling–Gupta efficiency (KGE) was increased by approximately 200%. The approach developed in this study establishes a comprehensive and scalable data processing and analysis pipeline that can be applied to diverse datasets and regions encountering similar domain-specific challenges.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Precipitation Measurement and Analysis · Meteorological Phenomena and Simulations · Soil Moisture and Remote Sensing
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Civil Engineering, Faculty of Engineering, University of Malaya, Kuala Lumpur 50603, Malaysia
Department of Environmental Science and Technology, University of Maryland, College Park, MD 20742, USA
Lee Kong Chian Faculty of Engineering and Science, Universiti Tunku Abdul Rahman, Kajang 43000, Malaysia
Faculty of Science and Engineering, University of Nottingham Malaysia, Semenyih 43500, Malaysia
References (89)
Show source-provided references
1. Precipitation recycling10.1029/96RG01927 ↗
2. Satellite rainfall climatology: A review10.1002/joc.635 ↗
3. So, How Much of the Earth’s Surface Is Covered by Rain Gauges?10.1175/BAMS-D-14-00283.1 ↗
4. Performance evaluation and correction of precipitation data using the 20-year IMERG and TMPA precipitation products in diverse subregions of China10.1016/j.atmosres.2020.105304 ↗
5. Comprehensive evaluation of multi-satellite precipitation products with a dense rain gauge network and optimally merging their simulated hydrological flows using the Bayesian model averaging method10.1016/j.jhydrol.2012.05.055 ↗
6. Comparison and validation of eight satellite rainfall products over the rugged topography of Tekeze-Atbara Basin at different spatial and temporal scales
7. Turini, N., Thies, B., and Bendix, J. (2019). Estimating High Spatio-Temporal Resolution Rainfall from MSG1 and GPM IMERG Based on Machine Learning: Case Study of Iran. Remote Sens., 11.10.3390/rs11192307 ↗
8. Hong, Y., Zhang, Y., and Khan, S. (2016). Hydrologic Remote Sensing: Capacity Building for Sustainability and Resilience, CRC Press.10.1201/9781315370392 ↗
9. Llauca, H., Lavado-Casimiro, W., León, K., Jimenez, J., Traverso, K., and Rau, P. (2021). Assessing Near Real-Time Satellite Precipitation Products for Flood Simulations at Sub-Daily Scales in a Sparsely Gauged Watershed in Peruvian Andes. Remote Sens., 13.10.3390/rs13040826 ↗
10. Near real-time hurricane rainfall forecasting using convolutional neural network models with Integrated Multi-satellitE Retrievals for GPM (IMERG) product10.1016/j.atmosres.2022.106037 ↗
11. Assessment of satellite precipitation product estimates over Bali Island10.1016/j.atmosres.2020.105032 ↗
12. The TRMM multisatellite precipitation analysis (TMPA): Quasi-global, multiyear, combined-sensor precipitation estimates at fine scales10.1175/JHM560.1 ↗
13. CMORPH: A method that produces global precipitation estimates from passive microwave and infrared data at high spatial and temporal resolution10.1175/1525-7541(2004)005<0487:CAMTPG>2.0.CO;2 ↗
14. Precipitation estimation from remotely sensed information using artificial neural networks10.1175/1520-0450(1997)036<1176:PEFRSI>2.0.CO;2 ↗
15. Evaluation of PERSIANN system satellite-based estimates of tropical rainfall10.1175/1520-0477(2000)081<2035:EOPSSE>2.3.CO;2 ↗
16. Global precipitation map using satellite-borne microwave radiometers by the GSMaP project: Production and validation10.1109/TGRS.2007.895337 ↗
17. The climate hazards infrared precipitation with stations—A new environmental record for monitoring extremes10.1038/sdata.2015.66 ↗
18. Huffman, G.J., Bolvin, D.T., Braithwaite, D., Hsu, K.-L., Joyce, R.J., Kidd, C., Nelkin, E.J., Sorooshian, S., Stocker, E.F., and Tan, J. (2020). Satellite Precipitation Measurement, Springer.
19. A two-stage blending approach for merging multiple satellite precipitation estimates and rain gauge observations: An experiment in the northeastern Tibetan Plateau10.5194/hess-25-359-2021 ↗
20. Evaluation of GPM-IMERG and TRMM-3B42 precipitation products over Pakistan10.1016/j.atmosres.2020.105341 ↗
21. A comprehensive evaluation of GPM-IMERG V06 and MRMS with hourly ground-based precipitation observations across Canada10.1016/j.jhydrol.2020.125929 ↗
22. Spatial evaluation of satellite-retrieved extreme rainfall rates in the Upper Awash River Basin, Ethiopia10.1016/j.atmosres.2020.105297 ↗
23. Huang, W.-R., Liu, P.-Y., Hsu, J., Li, X., and Deng, L. (2021). Assessment of Near-Real-Time Satellite Precipitation Products from GSMaP in Monitoring Rainfall Variations over Taiwan. Remote Sens., 13.10.3390/rs13020202 ↗
24. Comprehensive validation of the latest IMERG V06 precipitation estimates over a basin coupled with coastal locations, tropical climate and hill-karst combined landform10.1016/j.atmosres.2020.105293 ↗
25. Nepal, B., Shrestha, D., Sharma, S., Shrestha, M.S., Aryal, D., and Shrestha, N. (2021). Assessment of GPM-Era Satellite Products’ (IMERG and GSMaP) Ability to Detect Precipitation Extremes over Mountainous Country Nepal. Atmosphere, 12.10.3390/atmos12020254 ↗
26. Data Series Management: The Road to Big Sequence Analytics10.1145/2814710.2814719 ↗
27. Climate Engine: Cloud Computing and Visualization of Climate and Remote Sensing Data for Advanced Natural Resource Monitoring and Process Understanding10.1175/BAMS-D-15-00324.1 ↗
28. Blending multi-satellite, atmospheric reanalysis and gauge precipitation products to facilitate hydrological modelling10.1016/j.jhydrol.2020.125878 ↗
29. Google Earth Engine Cloud Computing Platform for Remote Sensing Big Data Applications: A Comprehensive Review10.1109/JSTARS.2020.3021052 ↗
30. A Machine Learning System for Precipitation Estimation Using Satellite and Ground Radar Network Observations10.1109/TGRS.2019.2942280 ↗
31. Merging multiple satellite-based precipitation products and gauge observations using a novel double machine learning approach10.1016/j.jhydrol.2021.125969 ↗
32. Improving near real-time precipitation estimation using a U-Net convolutional neural network and geographical information10.1016/j.envsoft.2020.104856 ↗
33. Improving the accuracy of rainfall rates from optical satellite sensors with machine learning—A random forests-based approach applied to MSG SEVIRI10.1016/j.rse.2013.10.026 ↗
34. and Lessmann, S. A comparative study of LSTM neural networks in forecasting day-ahead global horizontal irradiance with satellite data10.1016/j.solener.2018.01.005 ↗
35. Graves, A., Jaitly, N., and Mohamed, A.R. (2013, January 8–12). Hybrid speech recognition with deep bidirectional LSTM. Proceedings of the 2013 IEEE Workshop on Automatic Speech Recognition and Understanding, Olomouc, Czech Republic.10.1109/ASRU.2013.6707742 ↗
36. Wang, S., and Jiang, J. (2015). Learning natural language inference with LSTM. arXiv.10.18653/v1/N16-1170 ↗
37. Transductive LSTM for time-series prediction: An application to weather forecasting10.1016/j.neunet.2019.12.030 ↗
38. Zhou, S.K., Rueckert, D., and Fichtinger, G. (2020). Handbook of Medical Image Computing and Computer Assisted Intervention, Academic Press.
39. Fundamentals of recurrent neural network (RNN) and long short-term memory (LSTM) network10.1016/j.physd.2019.132306 ↗
40. Short-Term Precipitation Forecast Based on the PERSIANN System and LSTM Recurrent Neural Networks
41. Miao, Q., Pan, B., Wang, H., Hsu, K., and Sorooshian, S. (2019). Improving Monsoon Precipitation Prediction Using Combined Convolutional and Long Short Term Memory Neural Network. Water, 11.10.3390/w11050977 ↗
42. A spatiotemporal deep fusion model for merging satellite and gauge precipitation in China10.1016/j.jhydrol.2020.124664 ↗
43. Long Short-Term Memory10.1162/neco.1997.9.8.1735 ↗
44. Comparative evaluation of LSTM, CNN, and ConvLSTM for hourly short-term streamflow forecasting using deep learning approaches10.1016/j.ecoinf.2023.102119 ↗
45. Sharma, O. (2019, January 14–16). Deep challenges associated with deep learning. Proceedings of the 2019 International Conference on Machine Learning, Big Data, Cloud and Parallel Computing (COMITCon), Faridabad, India.10.1109/COMITCon.2019.8862453 ↗
46. Gao, W., Gao, J., Yang, L., Wang, M., and Yao, W. (2021). A novel modeling strategy of weighted mean temperature in China using RNN and LSTM. Remote Sens., 13.10.3390/rs13153004 ↗
47. Learning precise timing with LSTM recurrent networks
48. Noh, S.-H. (2021). Analysis of gradient vanishing of RNNs and performance comparison. Information, 12.10.3390/info12110442 ↗
49. Mirzaei, M., Yu, H., Dehghani, A., Galavi, H., Shokri, V., Karimi, S.M., and Sookhak, M. (2021). A novel stacked long short-term memory approach of deep learning for streamflow simulation. Sustainability, 13.10.3390/su132313384 ↗
50. Hybrids of machine learning techniques and wavelet regression for estimation of daily solar radiation10.1007/s00477-022-02261-8 ↗
51. A Survey on Data Collection for Machine Learning: A Big Data—AI Integration Perspective10.1109/TKDE.2019.2946162 ↗
52. Merging gauge and satellite rainfall with specification of associated uncertainty across Australia10.1016/j.jhydrol.2013.06.039 ↗
53. Sub-hourly resolution quality control of rain-gauge data significantly improves regional sub-daily return level estimates10.1002/qj.4357 ↗
54. Hsu, J., Huang, W.-R., Liu, P.-Y., and Li, X. (2021). Validation of CHIRPS Precipitation Estimates over Taiwan at Multiple Timescales. Remote Sens., 13.10.3390/rs13020254 ↗
55. Machine learning-based error modeling to improve GPM IMERG precipitation product over the brahmaputra river basin10.3390/forecast2030014 ↗
56. Improving satellite rainfall estimation from MSG data in Northern Algeria by using a multi-classifier model based on machine learning10.1016/j.jhydrol.2020.124705 ↗
57. Trends in rainfall and rainfall-related extremes in the east coast of peninsular Malaysia10.1007/s12040-015-0639-9 ↗
58. Numerical case study of an extreme rainfall event during 9–11 December 2004 over the east coast of Peninsular Malaysia10.1007/s00703-006-0236-1 ↗
59. Extreme rainstorms that caused devastating flooding across the east coast of Peninsular Malaysia during November and December 201410.1175/WAF-D-16-0160.1 ↗
60. Svennerberg, G. (2010). Beginning Google Maps API 3, Apress.10.1007/978-1-4302-2803-5 ↗
61. Method for Measuring the Information Content of Terrain from Digital Elevation Models10.3390/e17107021 ↗
62. Collette, A. (2013). Python and HDF5: Unlocking Scientific Data, O’Reilly Media, Inc.
63. Van Rossum, G., and Drake, F. (2009). Python 3 Reference Manual Createspace, CreateSpace.
64. Witham, M., Bender, I., and Gomes, R. (2019, January 5–6). Comparative Analysis of MariaDB’s Performance Efficiency as a Suitable Replacement for MySQL. Proceedings of the 2019 Midwest Instruction and Computing Symposiu, Fargo, ND, USA.
65. Lindstrom, J., Das, D., Mathiasen, T., Arteaga, D., and Talagala, N. (2015, January 19–21). NVM aware MariaDB database system. Proceedings of the 2015 IEEE Non-Volatile Memory System and Applications Symposium (NVMSA), Hong Kong, China.10.1109/NVMSA.2015.7304362 ↗
66. Jamison, D.C. (2003). Structured Query Language (SQL) Fundamentals. Curr. Protoc. Bioinform., 9.2.1–9.2.29.10.1002/0471250953.bi0902s00 ↗
67. Evaluation of satellite precipitation products for extreme flood events: Case study in Peninsular Malaysia10.2166/wcc.2018.159 ↗
68. Rainfall in the Greater and Lesser Antilles: Performance of five gridded datasets on a daily timescale10.1016/j.ejrh.2022.101203 ↗
69. Thirteen Ways to Look at the Correlation Coefficient
70. Enhanced identification of a hydrologic model using streamflow and satellite water storage data: A multicriteria sensitivity analysis and optimization approach10.1002/hyp.11267 ↗
71. Adjustment of global precipitation data for enhanced hydrologic modeling of tropical Andean watersheds10.1007/s10584-016-1706-1 ↗
72. Components of information for multiple resolution comparison between maps that share a real variable10.1007/s10651-007-0043-y ↗
73. On the use of dimensioned measures of error to evaluate the performance of spatial interpolators10.1080/13658810500286976 ↗
74. Another look at measures of forecast accuracy10.1016/j.ijforecast.2006.03.001 ↗
75. Decomposition of the mean squared error and NSE performance criteria: Implications for improving hydrological modelling10.1016/j.jhydrol.2009.08.003 ↗
76. Runoff conditions in the upper Danube basin under an ensemble of climate change scenarios10.1016/j.jhydrol.2012.01.011 ↗
77. Sak, H., Senior, A., and Beaufays, F. (2014). Long short-term memory based recurrent neural network architectures for large vocabulary speech recognition. arXiv.10.21437/Interspeech.2014-80 ↗
78. Predicting crop yields using a new robust Bayesian averaging model based on multiple hybrid ANFIS and MLP models10.1016/j.asej.2022.101724 ↗
79. Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv.
80. Bakken, S.S., Suraski, Z., and Schmid, E. (2000). PHP Manual: Volume 2, IUniverse, Incorporated.
81. An Experiment of Animation Development in Hypertext Preprocessor (PHP) and Hypertext Markup Language (HTML)
82. Random walk method for modeling water exchange: An application to coastal zone environmental management10.1016/j.jher.2015.07.001 ↗
83. Evaluation of spatial interpolation methods and spatiotemporal modeling of rainfall distribution in Peninsular Malaysia10.1016/j.asej.2021.09.001 ↗
84. Guo, B., Xu, T., Yang, Q., Zhang, J., Dai, Z., Deng, Y., and Zou, J. (2023). Multiple Spatial and Temporal Scales Evaluation of Eight Satellite Precipitation Products in a Mountainous Catchment of South China. Remote Sens., 15.10.3390/rs15051373 ↗
85. Correcting the bias of daily satellite precipitation estimates in tropical regions using deep neural network10.1016/j.jhydrol.2022.127656 ↗
86. Comparison of four machine learning algorithms for their applicability in satellite-based optical rainfall retrievals10.1016/j.atmosres.2015.09.021 ↗
87. Convolutional lstm architecture for precipitation nowcasting using satellite data
88. Review of deep learning: Concepts, CNN architectures, challenges, applications, future directions10.1186/s40537-021-00444-8 ↗
89. Forecasting of extreme flood events using different satellite precipitation products and wavelet-based machine learning methods10.1063/5.0008195 ↗