Semi-supervised Learning Remote Sensing Image Retrieval Method Based on Triplet Sampling Graph Convolutional Network

FENG Xiaoxin; WANG Zijian; WU Qi

doi:10.11999/JEIT211478

Volume 45 Issue 2

Feb. 2023

Turn off MathJax

Article Contents

Article Navigation > Journal of Electronics & Information Technology > 2023 > 45(2): 644-653

FENG Xiaoxin, WANG Zijian, WU Qi. Semi-supervised Learning Remote Sensing Image Retrieval Method Based on Triplet Sampling Graph Convolutional Network[J]. Journal of Electronics & Information Technology, 2023, 45(2): 644-653. doi: 10.11999/JEIT211478

Citation:

FENG Xiaoxin, WANG Zijian, WU Qi. Semi-supervised Learning Remote Sensing Image Retrieval Method Based on Triplet Sampling Graph Convolutional Network[J]. Journal of Electronics & Information Technology, 2023, 45(2): 644-653. doi: 10.11999/JEIT211478

Citation:

PDF( 5266 KB)

Semi-supervised Learning Remote Sensing Image Retrieval Method Based on Triplet Sampling Graph Convolutional Network

doi: 10.11999/JEIT211478

1.
School of Electronic, Information and Electrical Engineering, Shanghai Jiao Tong University, Shanghai 200240, China
2.
Department of Chemistry, The University of Hong Kong, Hong Kong 999077, China

Funds: The National Natural Science Foundation of China (U1933125)

Received Date: 2019-11-20
Rev Recd Date: 2022-04-28

Available Online: 2022-06-28

Publish Date: 2023-02-07

Abstract

Abstract

In this paper, a novel metric learning method based on the triplet sampling graph convolutional network is proposed to realize semi-supervised Content-Based Image Retrieval (CBIR) for remote sensing images. The proposed method consists of two parts: Triplet Graph Convolutional Network (TGCN) and Graph-based Triplet Sampling (GTS). TGCN is composed of three parallel convolutional neural networks and graph convolutional networks with shared weights to extract the initial features of the image and learn the graph embedding of the image. By learning simultaneously image features and graph embedding, TGCN can obtain an effective graph structure for semi-supervised image retrieval.Besides, the image similarity information implicit in the graph structure is evaluated by the proposed GTS algorithm to select the appropriate Hard triplet, and the sample set composed of the Hard triplet then can be used to train effectively and quickly the model. Through the combination of TGCN and GTS, the proposed metric learning method is tested on two remote sensing data sets. Experimental results show that TGCN-GTS has the following two advantages: TGCN can learn effective graph embedding features and metric space according to the image and graph structure; GTS evaluates effectively the image similarity information implicit in the image structure and then selects the appropriate Hard triplet, which improves significantly the retrieval performance of semi-supervised remote sensing images.
- Remote sensing image retrieval,
- Metric learning,
- Graph neural network,
- Semi-supervised learning

FullText(HTML)

References(30)

References

[1]	SUMBUL G, KANG Jian, and DEMIR B. Deep learning for image search and retrieval in large remote sensing archives[M]. CAMPS-VALLS G, TUIA D, ZHU Xiaoxiang, et al. Deep Learning for the Earth Sciences: A Comprehensive Approach to Remote Sensing, Climate Science, and Geosciences. Hoboken: John Wiley & Sons Ltd, 2021: 150–160.
[2]	YE Famao, XIAO Hui, ZHAO Xuqing, et al. Remote sensing image retrieval using convolutional neural network features and weighted distance[J]. IEEE Geoscience and Remote Sensing Letters, 2018, 15(10): 1535–1539. doi: 10.1109/LGRS.2018.2847303
[3]	ROY S, SANGINETO E, DEMIR B, et al. Metric-learning-based deep hashing network for content-based retrieval of remote sensing images[J]. IEEE Geoscience and Remote Sensing Letters, 2021, 18(2): 226–230. doi: 10.1109/LGRS.2020.2974629
[4]	SCHROFF F, KALENICHENKO D, and PHILBIN J. FaceNet: A unified embedding for face recognition and clustering[C]. Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition, Boston, USA, 2015: 815–823.
[5]	ZHU Panpan, TAN Yumin, ZHANG Liqiang, et al. Deep learning for multilabel remote sensing image annotation with dual-level semantic concepts[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 58(6): 4047–4060. doi: 10.1109/TGRS.2019.2960466
[6]	ZHANG Shizhou, ZHANG Qi, WEI Xing, et al. Person re-identification with triplet focal loss[J]. IEEE Access, 2018, 6: 78092–78099. doi: 10.1109/ACCESS.2018.2884743
[7]	KIM S, SEO M, LAPTEV I, et al. Deep metric learning beyond binary supervision[C]. Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, USA, 2019: 2283–2292.
[8]	WANG Xun, HAN Xintong, HUANG Weilin, et al. Multi-similarity loss with general pair weighting for deep metric learning[C]. Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, USA, 2019: 5017–5025.
[9]	WANG Xinshao, HUA Yang, KODIROV E, et al. Ranked list loss for deep metric learning[C]. Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, USA, 2019: 5202–5211.
[10]	ZHANG Maoding, CHENG Qimin, LUO Fang, et al. A triplet nonlocal neural network with dual-anchor triplet loss for high-resolution remote sensing image retrieval[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2021, 14: 2711–2723. doi: 10.1109/JSTARS.2021.3058691
[11]	WU Chaoyuan, MANMATHA R, SMOLA A J, et al. Sampling matters in deep embedding learning[C]. Proceedings of 2017 IEEE International Conference on Computer Vision, Venice, Italy, 2017: 2859–2867.
[12]	ZHANG Dingyi, LI Yingming, and ZHANG Zhongfei. Deep metric learning with spherical embedding[C]. Proceedings of the 34th International Conference on Neural Information Processing Systems, Vancouver, Canada, 2020: 1576.
[13]	XUAN Hong, STYLIANOU A, and PLESS R. Improved embeddings with easy positive triplet mining[C]. Proceedings of 2020 IEEE Winter Conference on Applications of Computer Vision, Snowmass, USA, 2020: 2463–2471.
[14]	YUAN Yuhui, YANG Kuiyuan, and ZHANG Chao. Hard-aware deeply cascaded embedding[C]. Proceedings of 2017 IEEE International Conference on Computer Vision, Venice, Italy, 2017: 814–823.
[15]	YANG Xun, ZHOU Peicheng, and WANG Meng. Person reidentification via structural deep metric learning[J]. IEEE Transactions on Neural Networks and Learning Systems, 2019, 30(10): 2987–2998. doi: 10.1109/TNNLS.2018.2861991
[16]	GE Weifeng, HUANG Weilin, DONG Dengke, et al. Deep metric learning with hierarchical triplet loss[C]. Proceedings of European Conference on Computer Vision, Munich, Germany, 2018: 272–288.
[17]	CAO Rui, ZHANG Qian, ZHU Jiasong, et al. Enhancing remote sensing image retrieval using a triplet deep metric learning network[J]. International Journal of Remote Sensing, 2020, 41(2): 740–751. doi: 10.1080/2150704X.2019.1647368
[18]	JIANG Bo, ZHANG Ziyan, LIN Doudou, et al. Semi-supervised learning with graph learning-convolutional networks[C]. Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, USA, 2019: 11305–11312.
[19]	DUVENAUD D, MACLAURIN D, AGUILERA-IPARRAGUIRRE J, et al. Convolutional networks on graphs for learning molecular fingerprints[C]. Proceedings of the 28th International Conference on Neural Information Processing Systems, Montreal, Canada, 2015: 2224–2232.
[20]	BRUNA J, ZAREMBA W, SZLAM A, et al. Spectral networks and locally connected networks on graphs[C]. Proceedings of the 2nd International Conference on Learning Representations, Banff, Canada, 2013.
[21]	KIPF T N and WELLING M. Semi-supervised classification with graph convolutional networks[C]. Proceedings of the 5th International Conference on Learning Representations, Toulon, France, 2017.
[22]	KANG Jian, FERNANDEZ-BELTRAN R, HONG Danfeng, et al. Graph relation network: Modeling relations between scenes for multilabel remote-sensing image classification and retrieval[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 59(5): 4355–4369. doi: 10.1109/TGRS.2020.3016020
[23]	CHAUDHURI U, BANERJEE B, and BHATTACHARYA A. Siamese graph convolutional network for content based remote sensing image retrieval[J]. Computer Vision and Image Understanding, 2019, 184: 22–30. doi: 10.1016/j.cviu.2019.04.004
[24]	RIBA P, FISCHER A, LLADÓS J, et al. Learning graph edit distance by graph neural networks[J]. Pattern Recognition, 2021, 120: 108132. doi: 10.1016/j.patcog.2021.108132
[25]	XIA Guisong, HU Jingwen, HU Fan, et al. AID: A benchmark data set for performance evaluation of aerial scene classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2017, 55(7): 3965–3981. doi: 10.1109/TGRS.2017.2685945
[26]	CHENG Gong, HAN Junwei, and LU Xiaoqiang. Remote sensing image scene classification: Benchmark and state of the art[J]. Proceedings of the IEEE, 2017, 105(10): 1865–1883. doi: 10.1109/JPROC.2017.2675998
[27]	HE Kaiming, ZHANG Xiangyu, REN Shaoqing, et al. Deep residual learning for image recognition[C]. Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, 2016: 770–778.
[28]	CHENG Gong, YANG Ceyuan, YAO Xiwen, et al. When deep learning meets metric learning: Remote sensing image scene classification via learning discriminative CNNs[J]. IEEE Transactions on Geoscience and Remote Sensing, 2018, 56(5): 2811–2821. doi: 10.1109/TGRS.2017.2783902
[29]	KANG Jian, FERNANDEZ-BELTRAN R, YE Zhen, et al. Deep metric learning based on scalable neighborhood components for remote sensing scene characterization[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 58(12): 8905–8918. doi: 10.1109/TGRS.2020.2991657
[30]	KANG Jian, FERNÁNDEZ-BELTRÁN R, YE Zhen, et al. High-rankness regularized semi-supervised deep metric learning for remote sensing imagery[J]. Remote Sensing, 2020, 12(16): 2603. doi: 10.3390/rs12162603