A Multiscale Delayed Channel Attention Network-Based Method for Pansharpening

Dajiang Lei; Lang Xiao; Yujia Li; Hefeng Huang; Xiaoyu Chen; Tingting Zhou; Shixing Ou; Liping Zhang

doi:doi:10.11648/j.ajrs.20241201.11

Research Article |

| Peer-Reviewed

A Multiscale Delayed Channel Attention Network-Based Method for Pansharpening

Dajiang Lei, Lang Xiao, Yujia Li, Hefeng Huang, Xiaoyu Chen, Tingting Zhou, Shixing Ou, Liping Zhang^*

Published in American Journal of Remote Sensing (Volume 12, Issue 1)

Received: 15 December 2023 Accepted: 28 December 2023 Published: 21 January 2024

Views: Downloads:

Download PDF

Share This Article

Twitter
Linked In
Facebook

Abstract

Nowadays, remote sensing images are widely used in many fields. To obtain high-quality remote sensing images, remote sensing image fusion methods have attracted much attention. Although convolutional neural network-based pansharpening methods have good results, these methods focus on the forward propagation of the network, which cannot effectively seek the mapping relationship between images. Moreover, it is difficult to obtain global information due to the limitations of convolutional operations. In this paper, we propose a pansharpening method based on multiscale delayed channel attention networks. The method iteratively seeks to correlate high-resolution stage features with the original low-resolution multispectral image, providing a mechanism for error feedback to map the error of each stage. Meanwhile, it designs a multiscale feature fusion module to fuse feature information from different fields of view. The design of the delayed channel attention mechanism makes the network acquire the association relationship between low-frequency information and high-frequency information through adaptive learning, giving different weights to high-frequency information, and making it more flexible in dealing with different types of information. Finally, the feature aggregation module is used to generate fused images and adjust the corresponding feature information. The experimental results obtained from the Gaofen-2 and WorldView-2 experimental data show that the method achieves a significant improvement compared to the current fusion algorithms.

Published in	American Journal of Remote Sensing (Volume 12, Issue 1)
DOI	10.11648/j.ajrs.20241201.11
Page(s)	1-13
Creative Commons	This is an Open Access article, distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution and reproduction in any medium or format, provided the original work is properly cited.
Copyright	Copyright © The Author(s), 2024. Published by Science Publishing Group

Keywords

Pansharpening, Multiscale, Positive Feedback, Attention Mechanism

References

[1]	Vivone, G., Dalla Mura, M., Garzelli, A., Restaino, R., Scarpa, G., Ulfarsson, M.O., Alparone, L., Chanussot, J. A New Benchmark Based on Recent Advances in Multispectral Pansharpening: Revisiting Pansharpening With Classical and Emerging Pansharpening Methods. IEEE Geoscience and Remote Sensing Magazine. 2021, 9(1), 53-81. doi: 10.1109/MGRS.2020.3019315.
[2]	Yilmaz, C. S., Yilmaz, V., Gungor, O. A theoretical and practical survey of image fusion methods for multispectral pansharpening. Information Fusion. 2022, 79, 1-43. doi: 10.1016/J.INFFUS.2021.10.001.
[3]	Kwarteng, P., Chavez, A. Extracting spectral contrast in Landsat Thematic Mapper image data using selective principal component analysis. Photogrammetric Engineering and Remote Sensing. 1989, 55(3), 339-348. doi: 0099-1112/89/5503-339$02.25/0.
[4]	Rahmani, S., Strait, M., Merkurjev, D., Moeller, M., Wittman, T. An Adaptive IHS Pan-Sharpening Method. IEEE Geoscience and Remote Sensing Letters. 2010, 7(4), 746-750. doi: 10.1109/LGRS.2010.2046715.
[5]	Garzelli, A., Nencini, F., Capobianco, L. Optimal MMSE Pan Sharpening of Very High Resolution Multispectral Images. IEEE Transactions on Geoscience and Remote Sensing. 2008, 46(1), 228-236. doi: 10.1109/TGRS.2007.907604.
[6]	Choi, J., Yu, K., Kim, Y. A new adaptive component- substitution-based satellite image fusion by using partial replacement. IEEE Transactions on Geoscience and Remote Sensing. 2010, 49(1), 295-309. doi: 10.1109/TGRS.2010.2051674.
[7]	Shensa, M. J. The discrete wavelet transform: wedding the a trous and Mallat algorithms. IEEE Transactions on Signal Processing. 1992, 40(10), 2464-2482. doi: 10.1109/78.157290.
[8]	Otazu, X., Gonzalez-Audicana, M., Fors, O., Nunez, J. Introduction of sensor spectral response into image fusion methods. Application to wavelet-based methods. IEEE Transactions on Geoscience and Remote Sensing. 2005, 43(10), 2376-2385. doi: 10.1109/TGRS.2005.856106.
[9]	Palsson, F., Sveinsson, J. R., Ulfarsson, M. O., Benediktsson, J. A. Model-Based Fusion of Multi- and Hyperspectral Images Using PCA and Wavelets. IEEE Transactions on Geoscience and Remote Sensing. 2015, 53(5), 2652-2663. doi: 10.1109/TGRS.2014.2363477.
[10]	Palsson, F., Sveinsson, J. R., Ulfarsson, M. O. A New Pansharpening Algorithm Based on Total Variation. IEEE Geoscience and Remote Sensing Letters. 2014, 11(1), 318-322. doi: 10.1109/LGRS.2013.2257669
[11]	Fu, X., Lin, Z., Huang, Y., Ding, X. A Variational Pan- Sharpening With Local Gradient Constraints. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 2019; pp. 10265-10274. doi: 10.1109/CVPR.2019.01051.
[12]	Wang, X., Bai, S., Li, Z., Song, R., Tao, J. The PAN and MS Image Pansharpening Algorithm Based on Adaptive Neural Network and Sparse Representation in the NSST Domain. IEEE Access. 2019, 7, 52508-52521. doi: 10.1109/ACCESS.2019.2910656.
[13]	Tian, X., Chen, Y., Yang, C., Gao, X., Ma, J. A Variational Pansharpening Method Based on Gradient Sparse Representation. IEEE Signal Processing Letters. 2020, 27, 1180-1184. doi: 10.1109/LSP.2020.3007325.
[14]	Lang, W., Zhao, Z., Fang, S., Cao, Y., Wang, Y. Sparse representation-based detail-injection method for pan-sharpening. Journal of Applied Remote Sensing. 2020, 14(2), 026523. doi: 10.1117/1.JRS.14.026523.
[15]	Gharbia, R., Hassanien, A. E., El-Baz, A. H., Elhoseny, M., Gunasekaran, M. Multi-spectral and panchromatic image fusion approach using stationary wavelet transform and swarm flower pollination optimization for remote sensing applications. Future Generation Computer Systems. 2018, 88, 501-511. doi: 10.1016/j.future.2018.06.022.
[16]	Yilmaz, V., Yilmaz, C. S., Gungor, O. Genetic algorithm-based synthetic variable ratio image fusion. Geocarto International. 2019, 36(9), 1-18. doi: 10.1080/10106049.2019.1629649.
[17]	Masi, G., Cozzolino, D., Verdoliva, L., Scarpa, G. Pansharpening by Convolutional Neural Networks. Remote Sensing. 2016, 8(7), 594. doi: 10.3390/rs8070594.
[18]	Yang, J., Fu, X., Hu, Y., Huang, Y., Ding, X., Paisley, J. PanNet: A Deep Network Architecture for Pan- Sharpening. In 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy, 2017; pp. 1753- 1761. doi: 10.1109/ICCV.2017.193.
[19]	Xu, S., Zhang, J., Zhao, Z., Sun, K., Liu, J., Zhang, C. Deep Gradient Projection Networks for Pan-sharpening. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, 2021; pp. 1366-1375. doi: 10.1109/CVPR46437.2021.00142.
[20]	Xiang, Z., Xiao, L., Liu, P., Zhang, Y. A Multi-Scale Densely Deep Learning Method for Pansharpening. In IGARSS 2019-2019 IEEE International Geoscience and Remote Sensing Symposium, Yokohama, Japan, 2019; pp. 2786-2789. doi: 10.1109/IGARSS.2019.8898095.
[21]	Liu, Q., Zhou, H., Xu, Q., Liu, X., Wang, Y. PSGAN: A Generative Adversarial Network for Remote Sensing Image Pan-Sharpening. IEEE Transactions on Geoscience and Remote Sensing. 2021, 59(12), 10227- 10242. doi: 10.1109/TGRS.2020.3042974.
[22]	Ma, J., Yu, W., Chen, C., Liang, P., Guo, X., Jiang, J. Pan-GAN: an unsupervised pan-sharpening method for remote sensing image fusion. Information Fusion. 2020, 62, 110-120. doi: 10.1016/j.inffus.2020.04.006.
[23]	Wang, J., Shao, Z., Huang, X., Lu, T., Zhang, R., Ma, J. Pan-Sharpening Via High-Pass Modification Convolutional Neural Network. In 2021 IEEE International Conference on Image Processing (ICIP), Anchorage, AK, USA, 2021; pp. 1714-1718. doi: 10.1109/ICIP42928.2021.9506568.
[24]	Hu, J., Shen, L., Sun, G. Squeeze-and-Excitation Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2020, 42(8), 2011-2023. doi: 10.1109/TPAMI.2019.2913372.
[25]	Jaderberg, M., Simonyan, K., Zisserman, A., Kavukcuoglu, K. Spatial Transformer Networks. Advances in Neural Information Processing Systems. 2015, 28, 2015-2017. doi: 10.48550/arXiv.1506.02025.
[26]	Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y. Image super-resolution using very deep residual channel attention networks. In Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 2018; pp. 286-301. doi: 10.48550/arXiv.1807.02758.
[27]	Wang, X., Girshick, R., Gupta, A., He, K. Non- local Neural Networks. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 2018; pp. 7794-7803. doi: 10.1109/CVPR.2018.00813.
[28]	He, K., Zhang, X., Ren, S., Sun, J. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 2016; pp. 770-778. doi: 10.1109/CVPR.2016.90.
[29]	Simonyan, K., Zisserman, A. Very Deep Convolutional Networks for Large-Scale Image Recognition. IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences. 2014. doi: 10.48550/arXiv.1409.1556.
[30]	Woo, S., Park, J., Lee, J. Y., Kweon, I. S. CBAM: Convolutional Block Attention Module. In Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 2018; pp. 3-19. doi: 10.48550/arXiv.1807.06521.
[31]	Vivone, G., Alparone, L., Chanussot, J., Dalla Mura, M., Garzelli, A., Licciardi, G.A., Restaino, R., Wald, L. A Critical Comparison Among Pansharpening Algorithms. IEEE Transactions on Geoscience and Remote Sensing. 2015, 53(5), 2565-2586. doi: 10.1109/TGRS.2014.2361734.
[32]	Zhang, Y., Liu, C., Sun, M., Ou, Y. Pan-sharpening using an efficient bidirectional pyramid network. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57(8), 5549-5563. doi: 10.1109/TGRS.2019.2900419.
[33]	Garzelli, A., Nencini, F. Hypercomplex Quality Assessment of Multi/Hyperspectral Images. IEEE Geoscience and Remote Sensing Letters. 2009, 6(4), 662-665. doi: 10.1109/LGRS.2009.2022650.
[34]	Zhou, J., Civco, D.L., Silander, J. A wavelet transform method to merge Landsat TM and SPOT panchromatic data. International Journal of Remote Sensing. 1998, 19(4), 743-757. doi: 10.1080/014311698215973.
[35]	Alparone, L., Aiazzi, B., Baronti, S., Garzelli, A., Nencini, F., Selva, M. Multispectral and panchromatic data fusion assessment without reference. Photogrammetric Engineering and Remote Sensing. 2008, 74(2), 193-200. doi: 10.14358/PERS.74.2.193.

Cite This Article

Plain Text BibTeX RIS

APA Style

Lei, D., Xiao, L., Li, Y., Huang, H., Chen, X., et al. (2024). A Multiscale Delayed Channel Attention Network-Based Method for Pansharpening. American Journal of Remote Sensing, 12(1), 1-13. https://doi.org/10.11648/j.ajrs.20241201.11

Copy | Download

ACS Style

Lei, D.; Xiao, L.; Li, Y.; Huang, H.; Chen, X., et al. A Multiscale Delayed Channel Attention Network-Based Method for Pansharpening. Am. J. Remote Sens. 2024, 12(1), 1-13. doi: 10.11648/j.ajrs.20241201.11

Copy | Download

AMA Style

Lei D, Xiao L, Li Y, Huang H, Chen X, et al. A Multiscale Delayed Channel Attention Network-Based Method for Pansharpening. Am J Remote Sens. 2024;12(1):1-13. doi: 10.11648/j.ajrs.20241201.11

Copy | Download

@article{10.11648/j.ajrs.20241201.11,
  author = {Dajiang Lei and Lang Xiao and Yujia Li and Hefeng Huang and Xiaoyu Chen and Tingting Zhou and Shixing Ou and Liping Zhang},
  title = {A Multiscale Delayed Channel Attention Network-Based Method for Pansharpening},
  journal = {American Journal of Remote Sensing},
  volume = {12},
  number = {1},
  pages = {1-13},
  doi = {10.11648/j.ajrs.20241201.11},
  url = {https://doi.org/10.11648/j.ajrs.20241201.11},
  eprint = {https://article.sciencepublishinggroup.com/pdf/10.11648.j.ajrs.20241201.11},
  abstract = {Nowadays, remote sensing images are widely used in many fields. To obtain high-quality remote sensing images, remote sensing image fusion methods have attracted much attention. Although convolutional neural network-based pansharpening methods have good results, these methods focus on the forward propagation of the network, which cannot effectively seek the mapping relationship between images. Moreover, it is difficult to obtain global information due to the limitations of convolutional operations. In this paper, we propose a pansharpening method based on multiscale delayed channel attention networks. The method iteratively seeks to correlate high-resolution stage features with the original low-resolution multispectral image, providing a mechanism for error feedback to map the error of each stage. Meanwhile, it designs a multiscale feature fusion module to fuse feature information from different fields of view. The design of the delayed channel attention mechanism makes the network acquire the association relationship between low-frequency information and high-frequency information through adaptive learning, giving different weights to high-frequency information, and making it more flexible in dealing with different types of information. Finally, the feature aggregation module is used to generate fused images and adjust the corresponding feature information. The experimental results obtained from the Gaofen-2 and WorldView-2 experimental data show that the method achieves a significant improvement compared to the current fusion algorithms.},
 year = {2024}
}

Copy | Download

TY  - JOUR
T1  - A Multiscale Delayed Channel Attention Network-Based Method for Pansharpening
AU  - Dajiang Lei
AU  - Lang Xiao
AU  - Yujia Li
AU  - Hefeng Huang
AU  - Xiaoyu Chen
AU  - Tingting Zhou
AU  - Shixing Ou
AU  - Liping Zhang
Y1  - 2024/01/21
PY  - 2024
N1  - https://doi.org/10.11648/j.ajrs.20241201.11
DO  - 10.11648/j.ajrs.20241201.11
T2  - American Journal of Remote Sensing
JF  - American Journal of Remote Sensing
JO  - American Journal of Remote Sensing
SP  - 1
EP  - 13
PB  - Science Publishing Group
SN  - 2328-580X
UR  - https://doi.org/10.11648/j.ajrs.20241201.11
AB  - Nowadays, remote sensing images are widely used in many fields. To obtain high-quality remote sensing images, remote sensing image fusion methods have attracted much attention. Although convolutional neural network-based pansharpening methods have good results, these methods focus on the forward propagation of the network, which cannot effectively seek the mapping relationship between images. Moreover, it is difficult to obtain global information due to the limitations of convolutional operations. In this paper, we propose a pansharpening method based on multiscale delayed channel attention networks. The method iteratively seeks to correlate high-resolution stage features with the original low-resolution multispectral image, providing a mechanism for error feedback to map the error of each stage. Meanwhile, it designs a multiscale feature fusion module to fuse feature information from different fields of view. The design of the delayed channel attention mechanism makes the network acquire the association relationship between low-frequency information and high-frequency information through adaptive learning, giving different weights to high-frequency information, and making it more flexible in dealing with different types of information. Finally, the feature aggregation module is used to generate fused images and adjust the corresponding feature information. The experimental results obtained from the Gaofen-2 and WorldView-2 experimental data show that the method achieves a significant improvement compared to the current fusion algorithms.
VL  - 12
IS  - 1
ER  -

Copy | Download

Author Information

Dajiang Lei

College of Computer, Chongqing University of Posts and Telecommunications, Chongqing, China; Chongqing Key Laboratory of Image Cognition, Chongqing University of Posts and Telecommunications, Chongqing, China
Lang Xiao

College of Computer, Chongqing University of Posts and Telecommunications, Chongqing, China; Chongqing Key Laboratory of Image Cognition, Chongqing University of Posts and Telecommunications, Chongqing, China
Yujia Li

College of Computer, Chongqing University of Posts and Telecommunications, Chongqing, China; Chongqing Key Laboratory of Image Cognition, Chongqing University of Posts and Telecommunications, Chongqing, China
Hefeng Huang

College of Computer, Chongqing University of Posts and Telecommunications, Chongqing, China; Chongqing Key Laboratory of Image Cognition, Chongqing University of Posts and Telecommunications, Chongqing, China
Xiaoyu Chen

College of Computer, Chongqing University of Posts and Telecommunications, Chongqing, China
Tingting Zhou

College of Computer, Chongqing University of Posts and Telecommunications, Chongqing, China
Shixing Ou

College of Computer, Chongqing University of Posts and Telecommunications, Chongqing, China
Liping Zhang

Chongqing Key Laboratory of Image Cognition, Chongqing University of Posts and Telecommunications, Chongqing, China

Contact Email

Download PDF

Sections

Plain Text BibTeX RIS

APA Style

Lei, D., Xiao, L., Li, Y., Huang, H., Chen, X., et al. (2024). A Multiscale Delayed Channel Attention Network-Based Method for Pansharpening. American Journal of Remote Sensing, 12(1), 1-13. https://doi.org/10.11648/j.ajrs.20241201.11

Copy | Download

ACS Style

Lei, D.; Xiao, L.; Li, Y.; Huang, H.; Chen, X., et al. A Multiscale Delayed Channel Attention Network-Based Method for Pansharpening. Am. J. Remote Sens. 2024, 12(1), 1-13. doi: 10.11648/j.ajrs.20241201.11

Copy | Download

AMA Style

Lei D, Xiao L, Li Y, Huang H, Chen X, et al. A Multiscale Delayed Channel Attention Network-Based Method for Pansharpening. Am J Remote Sens. 2024;12(1):1-13. doi: 10.11648/j.ajrs.20241201.11

Copy | Download

@article{10.11648/j.ajrs.20241201.11,
  author = {Dajiang Lei and Lang Xiao and Yujia Li and Hefeng Huang and Xiaoyu Chen and Tingting Zhou and Shixing Ou and Liping Zhang},
  title = {A Multiscale Delayed Channel Attention Network-Based Method for Pansharpening},
  journal = {American Journal of Remote Sensing},
  volume = {12},
  number = {1},
  pages = {1-13},
  doi = {10.11648/j.ajrs.20241201.11},
  url = {https://doi.org/10.11648/j.ajrs.20241201.11},
  eprint = {https://article.sciencepublishinggroup.com/pdf/10.11648.j.ajrs.20241201.11},
  abstract = {Nowadays, remote sensing images are widely used in many fields. To obtain high-quality remote sensing images, remote sensing image fusion methods have attracted much attention. Although convolutional neural network-based pansharpening methods have good results, these methods focus on the forward propagation of the network, which cannot effectively seek the mapping relationship between images. Moreover, it is difficult to obtain global information due to the limitations of convolutional operations. In this paper, we propose a pansharpening method based on multiscale delayed channel attention networks. The method iteratively seeks to correlate high-resolution stage features with the original low-resolution multispectral image, providing a mechanism for error feedback to map the error of each stage. Meanwhile, it designs a multiscale feature fusion module to fuse feature information from different fields of view. The design of the delayed channel attention mechanism makes the network acquire the association relationship between low-frequency information and high-frequency information through adaptive learning, giving different weights to high-frequency information, and making it more flexible in dealing with different types of information. Finally, the feature aggregation module is used to generate fused images and adjust the corresponding feature information. The experimental results obtained from the Gaofen-2 and WorldView-2 experimental data show that the method achieves a significant improvement compared to the current fusion algorithms.},
 year = {2024}
}

Copy | Download

TY  - JOUR
T1  - A Multiscale Delayed Channel Attention Network-Based Method for Pansharpening
AU  - Dajiang Lei
AU  - Lang Xiao
AU  - Yujia Li
AU  - Hefeng Huang
AU  - Xiaoyu Chen
AU  - Tingting Zhou
AU  - Shixing Ou
AU  - Liping Zhang
Y1  - 2024/01/21
PY  - 2024
N1  - https://doi.org/10.11648/j.ajrs.20241201.11
DO  - 10.11648/j.ajrs.20241201.11
T2  - American Journal of Remote Sensing
JF  - American Journal of Remote Sensing
JO  - American Journal of Remote Sensing
SP  - 1
EP  - 13
PB  - Science Publishing Group
SN  - 2328-580X
UR  - https://doi.org/10.11648/j.ajrs.20241201.11
AB  - Nowadays, remote sensing images are widely used in many fields. To obtain high-quality remote sensing images, remote sensing image fusion methods have attracted much attention. Although convolutional neural network-based pansharpening methods have good results, these methods focus on the forward propagation of the network, which cannot effectively seek the mapping relationship between images. Moreover, it is difficult to obtain global information due to the limitations of convolutional operations. In this paper, we propose a pansharpening method based on multiscale delayed channel attention networks. The method iteratively seeks to correlate high-resolution stage features with the original low-resolution multispectral image, providing a mechanism for error feedback to map the error of each stage. Meanwhile, it designs a multiscale feature fusion module to fuse feature information from different fields of view. The design of the delayed channel attention mechanism makes the network acquire the association relationship between low-frequency information and high-frequency information through adaptive learning, giving different weights to high-frequency information, and making it more flexible in dealing with different types of information. Finally, the feature aggregation module is used to generate fused images and adjust the corresponding feature information. The experimental results obtained from the Gaofen-2 and WorldView-2 experimental data show that the method achieves a significant improvement compared to the current fusion algorithms.
VL  - 12
IS  - 1
ER  -

Copy | Download