Pansharpening Applications in Ecological and Environmental Monitoring Using an Attention Mechanism-Based Dual-Stream Cross-Modality Fusion Network
Source abstract
Abstract
Pansharpening is a critical technique in remote sensing, particularly in ecological and environmental monitoring, where it is used to integrate panchromatic (PAN) and multispectral (MS) images. This technique plays a vital role in assessing environmental changes, monitoring biodiversity, and supporting conservation efforts. While many current pansharpening methods primarily rely on PAN images, they often overlook the distinct characteristics of MS images and the cross-modal relationships between them. To address this limitation, the paper presents a Dual-Stream Cross-modality Fusion Network (DCMFN), designed to offer reliable data support for environmental impact assessment, ecological monitoring, and material optimization in nanotechnology. The proposed network utilizes an attention mechanism to extract features from both PAN and MS images individually. Additionally, a Cross-Modality Feature Fusion Module (CMFFM) is introduced to capture the complex interrelationships between PAN and MS images, enhancing the reconstruction quality of pansharpened images. This method not only boosts the spatial resolution but also maintains the richness of multispectral information. Through extensive experiments, the DCMFN demonstrates superior performance over existing methods on three remote sensing datasets, excelling in both objective evaluation metrics and visual quality.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Image Fusion Techniques · Image Enhancement Techniques · Remote-Sensing Image Classification
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Electronic Information, Wangjiang Campus, Sichuan University, Wuhou District, Chengdu 610065, China
Funding
Sichuan university
References (40)
Show source-provided references
1. Pansharpening and spatiotemporal image fusion method for remote sensing10.1088/2631-8695/ad3a34 ↗
2. Supervised Detail-Guided Multiscale State-Space Model for Pan-Sharpening
3. Nanotechnology and the environment: A European perspective10.1016/j.stam.2006.10.002 ↗
4. Kaur, J. (2024, January 28–39). Revolutionizing Pan Sharpening in Remote Sensing with Cutting-Edge Deep Learning Optimization. Proceedings of the 2024 Second International Conference on Intelligent Cyber Physical Systems and Internet of Things (ICoICI), Coimbatore, India.10.1109/ICoICI62503.2024.10696465 ↗
5. Wang, S., Zou, X., Li, K., Xing, J., Cao, T., and Tao, P. (2024). Towards robust pansharpening: A large-scale high-resolution multi-scene dataset and novel approach. Remote Sens., 16.10.3390/rs16162899 ↗
6. Review of the pansharpening methods for remote sensing images based on the idea of meta-analysis: Practical discussion and challenges10.1016/j.inffus.2018.05.006 ↗
7. Attention is all you need
8. Vision Transformer for Pansharpening10.1109/TGRS.2022.3168465 ↗
9. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv.
10. Fusion of satellite images of different spatial resolutions: Assessing the quality of resulting images
11. A Critical Comparison Among Pansharpening Algorithms10.1109/TGRS.2014.2361734 ↗
12. A Review of Multi-resolution Analysis (MRA) and Multi-geometric Analysis (MGA) Tools Used in the Fusion of Remote Sensing Images10.1007/s00034-019-01316-6 ↗
13. DRFormer: Learning Disentangled Representation for Pan-Sharpening via Mutual Information- Based Transformer
14. Hu, X., Jiang, J., Liu, X., and Ma, J. (2021, January 5–9). Zero-Shot Multi-Focus Image Fusion. Proceedings of the 2021 IEEE International Conference on Multimedia and Expo (ICME), Shenzhen, China.10.1109/ICME51207.2021.9428413 ↗
15. Masi, G., Cozzolino, D., Verdoliva, L., and Scarpa, G. (2016). Pansharpening by Convolutional Neural Networks. Remote Sens., 8.10.3390/rs8070594 ↗
16. Yang, J., Fu, X., Hu, Y., Huang, Y., Ding, X., and Paisley, J. (2017, January 22–29). PanNet: A Deep Network Architecture for Pan-Sharpening. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.193 ↗
17. Remote sensing image fusion based on two-stream fusion network10.1016/j.inffus.2019.07.010 ↗
18. Gradient Guided Pyramidal Convolution Residual Network with Interactive Connections for Pan-sharpening10.1080/01431161.2021.1945703 ↗
19. Cross-Modality Interaction Network for Pan-Sharpening
20. Zhou, H., Liu, Q., and Wang, Y. (2022, January 18–22). PanFormer: A Transformer Based Model for Pan-Sharpening. Proceedings of the 2022 IEEE International Conference on Multimedia and Expo (ICME), Taipei, Taiwan.10.1109/ICME52920.2022.9859770 ↗
21. Nie, Z., Chen, L., Jeon, S., and Yang, X. (2022). Spectral-Spatial Interaction Network for Multispectral Image and Panchromatic Image Fusion. Remote Sens., 14.10.3390/rs14164100 ↗
22. A Dual-Attention Transformer Network for Pansharpening10.1109/JSEN.2023.3300263 ↗
23. Shi, W., Caballero, J., Huszár, F., Totz, J., and Wang, Z. (July, January 26). Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.207 ↗
24. Yuhas, R.H., Goetz, A.F.H., and Boardman, J.W. (1992). Discrimination among semi-arid landscape endmembers using the Spectral Angle Mapper (SAM) algorithm. Environ. Sci., 126879175.
25. Analysis of stream water quality and estimation of nutrient load with the aid of Quick Bird remote sensing imagery10.1080/02626667.2012.683792 ↗
26. The impact of integrating WorldView-2 sensor and environmental variables in estimating plantation forest species aboveground biomass and carbon stocks in uMgeni Catchment, South Africa10.1016/j.isprsjprs.2016.06.017 ↗
27. Akumu, C.E., Amadi, E.O., and Dennis, S. (2021). Application of Drone and WorldView-4 Satellite Data in Mapping and Monitoring Grazing Land Cover and Pasture Quality: Pre- and Post-Flooding. Land, 10.10.3390/land10030321 ↗
28. Comparison of Pansharpening Algorithms: Outcome of the 2006 GRS-S Data Fusion Contest10.1109/TGRS.2007.904923 ↗
29. Classification of Pansharpened Urban Satellite Images10.1109/JSTARS.2011.2176467 ↗
30. Hypercomplex Quality Assessment of Multi/Hyperspectral Images10.1109/LGRS.2009.2022650 ↗
31. Quality assessment parameters of the images obtained with pansharpening methods10.1063/5.0241417 ↗
32. Pansharpening Quality Assessment Using the Modulation Transfer Functions of Instruments10.1109/TGRS.2009.2029094 ↗
33. Yuan, M., Zhao, T., Li, B., and Wei, X. (2023). Learning to Pan-sharpening with Memories of Spatial Details. arXiv.10.2139/ssrn.4535929 ↗
34. Loshchilov, I., and Hutter, F. (2017). Decoupled Weight Decay Regularization. arXiv.
35. MTF-tailored Multiscale Fusion of High-resolution MS and Pan Imagery10.14358/PERS.72.5.591 ↗
36. A New Benchmark Based on Recent Advances in Multispectral Pansharpening: Revisiting pansharpening with classical and emerging pansharpening methods10.1109/MGRS.2020.3019315 ↗
37. Context-driven fusion of high spatial and spectral resolution images based on oversampled multiresolution analysis10.1109/TGRS.2002.803623 ↗
38. A New Adaptive Component-Substitution-Based Satellite Image Fusion by Using Partial Replacement10.1109/TGRS.2010.2051674 ↗
39. A Multi-Scale and Multi-Depth Convolutional Neural Network for Remote Sensing Imagery Pan-Sharpening10.1109/JSTARS.2018.2794888 ↗
40. Wang, Y., Deng, L.J., Zhang, T.J., and Wu, X. (2021, January 20–24). SSconv: Explicit Spectral-to-Spatial Convolution for Pansharpening. Proceedings of the 29th ACM International Conference on Multimedia, Virtual.10.1145/3474085.3475600 ↗