Scale Adaptive Fusion Network for Multimodal Remote Sensing Data Classification

LIU Xiaomin; YU Mengjun; QIAO Zhenzhuang; WANG Haoyu; XING Changda

doi:10.11999/JEIT240178

Article Contents

Article Navigation > Journal of Electronics & Information Technology > 2022 >

LIU Xiaomin, YU Mengjun, QIAO Zhenzhuang, WANG Haoyu, XING Changda. Scale Adaptive Fusion Network for Multimodal Remote Sensing Data Classification[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT240178

Citation:

LIU Xiaomin, YU Mengjun, QIAO Zhenzhuang, WANG Haoyu, XING Changda. Scale Adaptive Fusion Network for Multimodal Remote Sensing Data Classification[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT240178

Citation:

PDF( 3321 KB)

Scale Adaptive Fusion Network for Multimodal Remote Sensing Data Classification

doi: 10.11999/JEIT240178

School of Information and Control Engineering, China University of Mining and Technology, Xuzhou 221116, China

Funds: The National Natural Science Foundation of China (62303468, 62303469), The Natural Science Foundation of Jiangsu Province (BK20221116, BK20221112), China Postdoctoral Science Foundation (2023M733757), The Excellent Post Doctorate Program of Jiangsu Province (2022ZB530)

Received Date: 2024-03-15
Rev Recd Date: 2024-07-01

Available Online: 2024-07-06

Abstract

Abstract

The multimodal fusion method can effectively improve the ground object classification accuracy by using the complementary characteristics of different modalities, which has become a research hotspot in the various fields in recent years. The existing multimodal fusion methods have been successfully applied to multi-source remote sensing classification tasks oriented to HyperSpectral Image (HSI) and Light Detection And Ranging (LiDAR). However, existing research still faces many challenges, including difficulty in capturing spatial dependencies among irregular ground objects and obtaining discriminative information in multimodal data. To address the above challenges, a Scale Adaptive Fusion Network (SAFN)is proposed in this paper, by integrating the fusion of multimodal, multiscale, and multiview features into a unified framework. First, a dynamic multiscale graph module is proposed to capture the complex spatial dependencies of ground object, enhancing the model’s adaptability to irregular and scale-dissimilar ground object. Second, the complementary properties of LiDAR and HSI are utilized to constrain ground object within the same spatial neighborhood to have similar feature representations, thereby acquiring discriminative remote sensing features. Then, a multimodal spatial-spectral graph fusion module is proposed to establish feature interactions among multimodal, multiscale, and multiview features, providing discriminative fusion features for classification tasks by capturing class-recognition information that can be shared among features. Finally, the fusion features are fed into a classifier to obtain class probability scores for predicting the ground object class. To verify the effectiveness of SAFN, experiments are conducted on three datasets (i.e., Houston, Trento, and MUUFL). The experimental results show that, SAFN achieved state-of-the-art performance in multi-source remote sensing data classification tasks when compared with existing mainstream methods.
- Feature fusion,
- HyperSpectral Image(HSI),
- Light Detection And Ranging(LiDAR),
- Classification,
- Graph learning

FullText(HTML)

References(31)

References

[1]	WANG Leiquan, ZHU Tongchuan, KUMAR N, et al. Attentive-adaptive network for hyperspectral images classification with noisy labels[J]. IEEE Transactions on Geoscience and Remote Sensing, 2023, 61: 5505514. doi: 10.1109/TGRS.2023.3254159.
[2]	HANG Renlong, LI Zhu, GHAMISI P, et al. Classification of hyperspectral and LiDAR data using coupled CNNs[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 58(7): 4939–4950. doi: 10.1109/TGRS.2020.2969024.
[3]	王成龙, 赵倩, 赵琰, 等. 基于深度可分离卷积的实时遥感目标检测算法[J]. 电光与控制, 2022, 29(8): 45–49. doi: 10.3969/j.issn.1671-637X.2022.08.009. WANG Chenglong, ZHAO Qian, ZHAO Yan, et al. A real-time remote sensing target detection algorithm based on depth separable convolution[J]. Electronics Optics & Control, 2022, 29(8): 45–49. doi: 10.3969/j.issn.1671-637X.2022.08.009.
[4]	AHMAD M, KHAN A M, MAZZARA M, et al. Multi-layer extreme learning machine-based autoencoder for hyperspectral image classification[C]. The 14th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, Prague, Czech, 2019: 75–82. doi: 10.5220/0007258000750082.
[5]	CUI Ying, SHAO Chao, LUO Li, et al. Center weighted convolution and GraphSAGE cooperative network for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2023, 61: 5508216. doi: 10.1109/TGRS.2023.3264653.
[6]	LI Mingsong, LI Wei, LIU Yikun, et al. Adaptive mask sampling and manifold to Euclidean subspace learning with distance covariance representation for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2023, 61: 5508518. doi: 10.1109/TGRS.2023.3265388.
[7]	OU Xianfeng, WU Meng, TU Bing, et al. Multi-objective unsupervised band selection method for hyperspectral images classification[J]. IEEE Transactions on Image Processing, 2023, 32: 1952–1965. doi: 10.1109/TIP.2023.3258739.
[8]	XUE Zhixiang, TAN Xiong, YU Xuchu, et al. Deep hierarchical vision transformer for hyperspectral and LiDAR data classification[J]. IEEE Transactions on Image Processing, 2022, 31: 3095–3110. doi: 10.1109/TIP.2022.3162964.
[9]	赵伍迪, 李山山, 李安, 等. 结合深度学习的高光谱与多源遥感数据融合分类[J]. 遥感学报, 2021, 25(7): 1489–1502. doi: 10.11834/jrs.20219117. ZHAO Wudi, LI Shanshan, LI An, et al. Deep fusion of hyperspectral images and multi-source remote sensing data for classification with convolutional neural network[J]. National Remote Sensing Bulletin, 2021, 25(7): 1489–1502. doi: 10.11834/jrs.20219117.
[10]	ZHAO Xudong, ZHANG Mengmeng, TAO Ran, et al. Fractional Fourier image transformer for multimodal remote sensing data classification[J]. IEEE Transactions on Neural Networks and Learning Systems, 2024, 35(2): 2314–2326. doi: 10.1109/TNNLS.2022.3189994.
[11]	ROY S K, DERIA A, HONG Danfeng, et al. Hyperspectral and LiDAR data classification using joint CNNs and morphological feature learning[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 5530416. doi: 10.1109/TGRS.2022.3177633.
[12]	孙强, 陈远. 多层次时空特征自适应集成与特有-共享特征融合的双模态情感识别[J]. 电子与信息学报, 2024, 46(2): 574–587. doi: 10.11999/JEIT231110. SUN Qiang and CHEN Yuan. Bimodal emotion recognition with adaptive integration of multi-level spatial-temporal features and specific-shared feature fusion[J]. Journal of Electronics & Information Technology. 2024, 46(2): 574–587. doi: 10.11999/JEIT231110.
[13]	雷大江, 杜加浩, 张莉萍, 等. 联合多流融合和多尺度学习的卷积神经网络遥感图像融合方法[J]. 电子与信息学报, 2022, 44(1): 237–244. doi: 10.11999/JEIT200792. LEI Dajiang, DU Jiahao, ZHANG Liping, et al. Multi-stream architecture and multi-scale convolutional neural network for remote sensing image fusion[J]. Journal of Electronics & Information Technology, 2022, 44(1): 237–244. doi: 10.11999/JEIT200792.
[14]	JIA Sen, ZHAN Zhangwei, ZHANG Meng, et al. Multiple feature-based superpixel-level decision fusion for hyperspectral and LiDAR data classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 59(2): 1437–1452. doi: 10.1109/TGRS.2020.2996599.
[15]	ZHAO Guangrui, YE Qiaolin, SUN Le, et al. Joint classification of hyperspectral and LiDAR data using a hierarchical CNN and transformer[J]. IEEE Transactions on Geoscience and Remote Sensing, 2023, 61: 5500716. doi: 10.1109/TGRS.2022.3232498.
[16]	LI Hengchao, HU Wenshuai, LI Wei, et al. A³ CLNN: Spatial, spectral and multiscale attention ConvLSTM neural network for multisource remote sensing data classification[J]. IEEE Transactions on Neural Networks and Learning Systems, 2022, 33(2): 747–761. doi: 10.1109/TNNLS.2020.3028945.
[17]	ZHANG Mengmeng, LI Wei, ZHANG Yuxiang, et al. Hyperspectral and LiDAR data classification based on structural optimization transmission[J]. IEEE Transactions on Cybernetics, 2023, 53(5): 3153–3164. doi: 10.1109/TCYB.2022.3169773.
[18]	LI Jiaojiao, MA Yinle, SONG Rui, et al. A triplet semisupervised deep network for fusion classification of hyperspectral and LiDAR data[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 5540513. doi: 10.1109/TGRS.2022.3213513.
[19]	DONG Wenqian, ZHANG Tian, QU Jiahui, et al. Multibranch feature fusion network with self- and cross-guided attention for hyperspectral and LiDAR classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 5530612. doi: 10.1109/TGRS.2022.3179737.
[20]	马梁, 苟于涛, 雷涛, 等. 基于多尺度特征融合的遥感图像小目标检测[J]. 光电工程, 2022, 49(4): 210363. doi: 10.12086/oee.2022.210363. MA Liang, GOU Yutao, LEI Tao, et al. Small object detection based on multi-scale feature fusion using remote sensing images[J]. Opto-Electronic Engineering, 2022, 49(4): 210363. doi: 10.12086/oee.2022.210363.
[21]	ZHANG Zhongqiang, LIU Danhua, GAO Dahua, et al. A novel spectral-spatial multi-scale network for hyperspectral image classification with the Res2Net block[J]. International Journal of Remote Sensing, 2022, 43(3): 751–777. doi: 10.1080/01431161.2021.2005840.
[22]	XU Kejie, ZHAO Yue, ZHANG Lingming, et al. Spectral–spatial residual graph attention network for hyperspectral image classification[J]. IEEE Geoscience and Remote Sensing Letters, 2022, 19: 5509305. doi: 10.1109/LGRS.2021.3111985.
[23]	TAN Xiong and XUE Zhixiang. Spectral-spatial multi-layer perceptron network for hyperspectral image land cover classification[J]. European Journal of Remote Sensing, 2022, 55(1): 409–419. doi: 10.1080/22797254.2022.2087540.
[24]	HONG Danfeng, HAN Zhu, YAO Jing, et al. SpectralFormer: Rethinking hyperspectral Image classification with transformers[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 5518615. doi: 10.1109/TGRS.2021.3130716.
[25]	HAMILTON W L, YING R, and LESKOVEC J. Inductive representation learning on large graphs[C]. The 31st International Conference on Neural Information Processing Systems, Long Beach, USA, 2017: 1025–1035.
[26]	YU Haoyang, ZHANG Hao, LIU Yao, et al. Dual-channel convolution network with image-based global learning framework for hyperspectral image classification[J]. IEEE Geoscience and Remote Sensing Letters, 2022, 19: 6005705. doi: 10.1109/LGRS.2021.3139358.
[27]	ZHAO Xudong, TAO Ran, LI Wei, et al. Joint classification of hyperspectral and LiDAR data using hierarchical random walk and deep CNN architecture[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 58(10): 7355–7370. doi: 10.1109/TGRS.2020.2982064.
[28]	HONG Danfeng, GAO Lianru, HANG Renlong, et al. Deep encoder–decoder networks for classification of hyperspectral and LiDAR data[J]. IEEE Geoscience and Remote Sensing Letters, 2022, 19: 5500205. doi: 10.1109/LGRS.2020.3017414.
[29]	WU Xin, HONG Danfeng, and CHANUSSOT J. Convolutional neural networks for multimodal remote sensing data classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 5517010. doi: 10.1109/TGRS.2021.3124913.
[30]	ZHAO Xudong, TAO Ran, LI Wei, et al. Fractional Gabor convolutional network for multisource remote sensing data classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 5503818. doi: 10.1109/TGRS.2021.3065507.
[31]	WANG Haoyu, CHENG Yuhu, LIU Xiaomin, et al. Reinforcement learning based Markov edge decoupled fusion network for fusion classification of hyperspectral and LiDAR[J]. IEEE Transactions on Multimedia, 2024, 26: 7174–7187. doi: 10.1109/TMM.2024.3360717.