A Continual Semantic Segmentation Method Based on Gating Mechanism and Replay Strategy

YANG Jing; HE Yao; LI Bin; LI Shaobo; HU Jianjun; PU Jiang

doi:10.11999/JEIT230803

Volume 46 Issue 7

Jul. 2024

Turn off MathJax

Article Contents

Article Navigation > Journal of Electronics & Information Technology > 2024 > 46(7): 2908-2917

YANG Jing, HE Yao, LI Bin, LI Shaobo, HU Jianjun, PU Jiang. A Continual Semantic Segmentation Method Based on Gating Mechanism and Replay Strategy[J]. Journal of Electronics & Information Technology, 2024, 46(7): 2908-2917. doi: 10.11999/JEIT230803

Citation:

YANG Jing, HE Yao, LI Bin, LI Shaobo, HU Jianjun, PU Jiang. A Continual Semantic Segmentation Method Based on Gating Mechanism and Replay Strategy[J]. Journal of Electronics & Information Technology, 2024, 46(7): 2908-2917. doi: 10.11999/JEIT230803

Citation:

PDF( 5238 KB)

A Continual Semantic Segmentation Method Based on Gating Mechanism and Replay Strategy

doi: 10.11999/JEIT230803

YANG Jing^{1, 2},
HE Yao^{1, 2
,
,},
LI Bin^{1, 2},
LI Shaobo^{1, 2},
HU Jianjun³,
PU Jiang⁴

1.
School of Mechanical Engineering, Guizhou University Guiyang 550025, China
2.
State Key Laboratory of Public Big Data, Guizhou University, Guiyang 550025, China
3.
Department of Computer Science and Engineering, University of South Carolina, Columbia 29208, USA
4.
School of Mechatronics Engineering, Guizhou Minzu University, Guiyang 550025, China

Funds: The National Natural Science Foundation of China (62166005), Developing Objects and Projects of Scientific and Technological Talents in Guiyang City (ZKH[2023]48-8), The National Key R&D Program of China (2018AAA010800)

Received Date: 2023-08-01
Rev Recd Date: 2023-12-02

Available Online: 2023-12-14

Publish Date: 2024-07-29

Abstract

Abstract

Due to the interference and background drift between new and old task parameters, semantic segmentation model based on deep neural networks promotes catastrophic forgetting of old knowledge. Furthermore, information frequently cannot be stored owing to privacy concerns, security concerns, and other issues, which leads to model failure. Therefore, a continual semantic segmentation method based on gating mechanism and replay strategy is proposed. First, without storing old data, generative adversarial network and webpage crawling are used as data sources, the label evaluation module is used to solve the unsupervised problem and the background self-drawing module is used to solve background drift problem. Then, catastrophic forgetting is mitigated by replay strategy; Finally, gated variables are used as a regularization means to increase the sparsity of the module and study the special case of gated variables combined with continual learning replay strategy. Our evaluation results on the Pascal VOC2012 dataset show that in the settings of complex scenario 10-2, Generative Adversarial Networks (GAN) and Web, the performance of the old task after all incremental steps are improved by 3.8% and 3.7% compared with the baseline, and in scenario 10-1, they are improved by 2.7% and 1.3% compared with the baseline, respectively.
- Continual learning,
- Semantic segmentation,
- Replay strategies,
- Gating variables

FullText(HTML)

References(28)

References

[1]	GONG Xuan, XIA Xin, ZHU Wentao, et al. Deformable Gabor feature networks for biomedical image classification[C]. 2021 IEEE Winter Conference on Applications of Computer Vision, Waikoloa, USA, 2021: 4003–4011. doi: 10.1109/WACV48630.2021.00405.
[2]	NING Xin, TIAN Weijuan, YU Zaiyang, et al. HCFNN: High-order coverage function neural network for image classification[J]. Pattern Recognition, 2022, 131: 108873. doi: 10.1016/j.patcog.2022.108873.
[3]	HE Junjun, DENG Zhongying, ZHOU Lei, et al. Adaptive pyramid context network for semantic segmentation[C]. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, USA, 2019: 7511–7520. doi: 10.1109/CVPR.2019.00770.
[4]	YANG Jing, LI Shaobo, WANG Zheng, et al. Using deep learning to detect defects in manufacturing: A comprehensive survey and current challenges[J]. Materials, 2020, 13(24): 5755. doi: 10.3390/ma13245755.
[5]	CHEN Pengfei, YU Xuehui, HAN Xumeng, et al. Point-to-box network for accurate object detection via single point supervision[C]. 17th European Conference on Computer Vision, Tel Aviv, Israel, 2022: 51–67. doi: 10.1007/978-3-031-20077-9_4.
[6]	SHENG Hualian, CAI Sijia, ZHAO Na, et al. Rethinking IoU-based optimization for single-stage 3D object detection[C]. 17th European Conference on Computer Vision, Tel Aviv, Israel, 2022: 544–561. doi: 10.1007/978-3-031-20077-9_32.
[7]	CHAUDHRY A, ROHRBACH M, ELHOSEINY M, et al. Continual learning with tiny episodic memories[EB/OL]. https://arxiv.org/abs/1902.10486v1, 2019.
[8]	KIRKPATRICK J, PASCANU R, RABINOWITZ N, et al. Overcoming catastrophic forgetting in neural networks[J]. Proceedings of the National Academy of Sciences of the United States of America, 2017, 114(13): 3521–3526. doi: 10.1073/pnas.1611835114.
[9]	ZENKE F, POOLE B, and GANGULI S. Continual learning through synaptic intelligence[C]. The 34th International Conference on Machine Learning, Sydney, Australia, 2017: 3987–3995.
[10]	ALJUNDI R, BABILONI F, ELHOSEINY M, et al. Memory aware synapses: Learning what (not) to forget[C]. The 15th European Conference on Computer Vision, Munich, Germany, 2018: 144–161. doi: 10.1007/978-3-030-01219-9_9.
[11]	VAN DE VEN G M and TOLIAS A S. Three scenarios for continual learning[EB/OL]. https://arxiv.org/abs/1904.07734, 2019.
[12]	WU Yue, CHEN Yinpeng, WANG Lijuan, et al. Large scale incremental learning[C]. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, USA, 2019: 374–382. doi: 10.1109/CVPR.2019.00046.
[13]	ZHAI Mengyao, CHEN Lei, and MORI G. Hyper-LifelongGAN: Scalable lifelong learning for image conditioned generation[C]. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, USA, 2021: 2246–2255. doi: 10.1109/CVPR46437.2021.00228.
[14]	GRAFFIETI G, MALTONI D, PELLEGRINI L, et al. Generative negative replay for continual learning[J]. Neural Networks, 2023, 162: 369–383. doi: 10.1016/j.neunet.2023.03.006.
[15]	MARACANI A, MICHIELI U, TOLDO M, et al. RECALL: Replay-based continual learning in semantic segmentation[C]. 2021 IEEE/CVF International Conference on Computer Vision, Montreal, Canada, 2021: 7006–7015. doi: 10.1109/ICCV48922.2021.00694.
[16]	CERMELLI F, MANCINI M, BULÒ S R, et al. Modeling the background for incremental learning in semantic segmentation[C]. x2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, USA, 2020: 9230–9239. doi: 10.1109/CVPR42600.2020.00925.
[17]	BALDI P and SADOWSKI P J. Understanding dropout[C]. The 26th International Conference on Neural Information Processing Systems, Lake Tahoe, USA, 2013: 2814–2822.
[18]	MIRZADEH S I, FARAJTABAR M, and GHASEMZADEH H. Dropout as an implicit gating mechanism for continual learning[C]. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Seattle, USA, 2020: 945–951. doi: 10.1109/CVPRW50498.2020.00124.
[19]	HE Kaiming, ZHANG Xiangyu, REN Shaoqing, et al. Deep residual learning for image recognition[C]. 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, 2016: 770–778. doi: 10.1109/CVPR.2016.90.
[20]	CHEN L C, PAPANDREOU G, KOKKINOS I, et al. DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(4): 834–848. doi: 10.1109/TPAMI.2017.2699184.
[21]	LI Zhizhong and HOIEM D. Learning without forgetting[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(12): 2935–2947. doi: 10.1109/TPAMI.2017.2773081.
[22]	REBUFFI S A, KOLESNIKOV A, SPERL G, et al. iCaRL: Incremental classifier and representation learning[C]. 2017 IEEE conference on Computer Vision and Pattern Recognition, Honolulu, USA, 2017: 5533–5542. doi: 10.1109/CVPR.2017.587.
[23]	MICHIELI U and ZANUTTIGH P. Incremental learning techniques for semantic segmentation[C]. 2019 IEEE/CVF International Conference on Computer Vision Workshop, Seoul, Korea (South), 2019: 3205–3212. doi: 10.1109/iccvw.2019.00400.
[24]	KLINGNER M, BÄR A, DONN P, et al. Class-incremental learning for semantic segmentation re-using neither old data nor old labels[C]. 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), Rhodes, Greece, 2020: 1–8. doi: 10.1109/ITSC45102.2020.9294483.
[25]	MICHIELI U and ZANUTTIGH P. Continual semantic segmentation via repulsion-attraction of sparse and disentangled latent representations[C]. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, USA, 2021: 1114–1124. doi: 10.1109/CVPR46437.2021.00117.
[26]	LI Junxi, SUN Xian, DIAO Wenhui, et al. Class-incremental learning network for small objects enhancing of semantic segmentation in aerial imagery[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 5612920. doi: 10.1109/TGRS.2021.3124303.
[27]	DOUILLARD A, CHEN Yifu, DAPOGNY A, et al. PLOP: Learning without forgetting for continual semantic segmentation[C]. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, USA, 2021: 4039–4049. doi: 10.1109/CVPR46437.2021.00403.
[28]	ZHAO Danpei, YUAN Bo, and SHI Zhenwei. Inherit with distillation and evolve with contrast: Exploring class incremental semantic segmentation without exemplar memory[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(10): 11932–11947. doi: 10.1109/TPAMI.2023.3273574.