Research Progress Analysis of Point Cloud Segmentation Based on Deep Learning

ZHAO Jiaqi; ZHOU Yong; HE Xin; BU Yifan; YAO Rui; GUO Rui

doi:10.11999/JEIT210972

Volume 44 Issue 12

Dec. 2022

Turn off MathJax

Article Contents

Article Navigation > Journal of Electronics & Information Technology > 2022 > 44(12): 4426-4440

ZHAO Jiaqi, ZHOU Yong, HE Xin, BU Yifan, YAO Rui, GUO Rui. Research Progress Analysis of Point Cloud Segmentation Based on Deep Learning[J]. Journal of Electronics & Information Technology, 2022, 44(12): 4426-4440. doi: 10.11999/JEIT210972

Citation:

ZHAO Jiaqi, ZHOU Yong, HE Xin, BU Yifan, YAO Rui, GUO Rui. Research Progress Analysis of Point Cloud Segmentation Based on Deep Learning[J]. Journal of Electronics & Information Technology, 2022, 44(12): 4426-4440. doi: 10.11999/JEIT210972

Citation:

PDF( 4413 KB)

Research Progress Analysis of Point Cloud Segmentation Based on Deep Learning

doi: 10.11999/JEIT210972

ZHAO Jiaqi^{1, 2, 3},
ZHOU Yong^{1, 2
,
,},
HE Xin^{1, 2},
BU Yifan¹,
YAO Rui^{1, 2},
GUO Rui^{1, 2}

1.
School of Computer Science and Technology, China University of Mining and Technology, Xuzhou 221116, China
2.
Ministry of Education Engineering Research Center of Mine Digitization, Xuzhou 221116, China
3.
Innovation Research Center of Disaster Intelligent Prevention and Emergency Rescue, Xuzhou 221116, China

Funds: The National Natural Science Foundation of China (61806206, 62172417), The Natural Science Foundation of Jiangsu Province (BK20180639, BK20201346), The Six Talent Peaks Project in Jiangsu Province (2015-DZXX-010, 2018-XYDXX-044)

Received Date: 2021-09-13
Accepted Date: 2021-12-06
Rev Recd Date: 2021-12-01

Available Online: 2021-12-06

Publish Date: 2022-12-16

Abstract

Abstract

The rapid development of depth sensor and laser scanning technology allows people to collect easily a large amount of point cloud data. Point cloud data can provide rich scene and object information, and has become the preferred research object for applications such as autonomous driving, virtual reality, and robot navigation. As an important research method, point cloud segmentation has received extensive attention from industry and academia. Especially driven by deep learning, the effect of point cloud segmentation has been significantly improved. In order to stimulate future research, the latest progress in point cloud segmentation are comprehensively reviewed , and comparative studies are conducted from the perspective of indirect and direct processing of point clouds. Among them, the method based on indirect processing can be divided into the multi-view and voxel-based method. The method based on direct processing can be divided into point processing, optimized convolutional neural networks , graph convolution, timing and unsupervised learning. Then the basic ideas and characteristics of the representative methods are introduced in each category. In addition, the common data sets and evaluation indexes of point cloud segmentation are sorted out. Finally, the future of point cloud classification and segmentation technology is prospected.
- Deep learning,
- Point cloud,
- Point cloud segmentation,
- Progress analysis

FullText(HTML)

References(67)

References

[1]	王肖锋, 张明路, 刘军. 基于增量式双向主成分分析的机器人感知学习方法研究[J]. 电子与信息学报, 2018, 40(3): 618–625. doi: 10.11999/JEIT170561 WANG Xiaofeng, ZHANG Minglu, and Liu Jun. Robot perceptual learning method based on incremental bidirectional principal component analysis[J]. Journal of Electronics &Information Technology, 2018, 40(3): 618–625. doi: 10.11999/JEIT170561
[2]	周汉飞, 李禹, 粟毅. 利用多角度SAR数据实现三维成像[J]. 电子与信息学报, 2013, 35(10): 2467–2474. doi: 10.3724/SP.J.1146.2012.01534 ZHOU Hanfei, LI Yu, and SU Yi. Three-dimensional imaging with multi-aspect SAR data[J]. Journal of Electronics &Information Technology, 2013, 35(10): 2467–2474. doi: 10.3724/SP.J.1146.2012.01534
[3]	QI C R, LIU Wei, WU Chenxia, et al. Frustum pointnets for 3D object detection from RGB-D data[C]. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, USA, 2018: 918–927.
[4]	陈境焕, 李海艳, 林景亮. 基于深度学习的零件点云分割算法研究[J]. 机电工程, 2020, 37(3): 326–331. doi: 10.3969/j.issn.1001-4551.2020.03.020 CHEN Jinghuan, LI Haiyan, and LIN Jingliang. Part point cloud segmentation algorithm based on deep learning[J]. Mechanical &Electrical Engineering, 2020, 37(3): 326–331. doi: 10.3969/j.issn.1001-4551.2020.03.020
[5]	KLOKOV R and LEMPITSKY V. Escape from cells: Deep kd-networks for the recognition of 3D point cloud models[C]. Proceedings of 2017 IEEE International Conference on Computer Vision, Venice, Italy, 2017: 863–872.
[6]	CHEN Xuzhan, CHEN Youping, and NAJJARAN H. 3D object classification with point convolution network[C]. 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems, Vancouver, Canada, 2017: 783–788.
[7]	HU Qingyong, YANG Bo, XIE Linhai, et al. RandLA-Net: Efficient semantic segmentation of large-scale point clouds[C]. Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, USA, 2020: 11105–11114.
[8]	CHEN Xiaozhi, MA Huimin, WAN Ji, et al. Multi-view 3D object detection network for autonomous driving[C]. 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, USA, 2017: 6526–6534.
[9]	ZHOU Dingfu, FANG Jin, SONG Xibin, et al. Joint 3D instance segmentation and object detection for autonomous driving[C]. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, USA, 2020: 1836–1846.
[10]	QI C R, SU Hao, MO Kaichun, et al. Pointnet: Deep learning on point sets for 3D classification and segmentation[C]. 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, USA, 2017: 77–85.
[11]	LI Ruihui, LI Xianzhi, HENG P A, et al. PointAugment: An auto-augmentation framework for point cloud classification[C]. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, USA, 2020: 6377–6386.
[12]	GADELHA M, WANG Rui, and MAJI S. Multiresolution tree networks for 3D point cloud processing[C]. The 15th European Conference on Computer Vision, Munich, Germany, 2018: 105–122.
[13]	GRAHAM B, ENGELCKE M, and VAN DER MAATEN L. 3D semantic segmentation with submanifold sparse convolutional networks[C]. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, USA, 2018: 9224–9232.
[14]	HUANG Qiangui, WANG Weiyue, and NEUMANN U. Recurrent slice networks for 3D segmentation of point clouds[C]. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, USA, 2018: 2626–2635.
[15]	KUMAR K S C and AL-STOUHI S. Multi-scale voxel class balanced ASPP for LIDAR pointcloud semantic segmentation[C]. 2021 IEEE Winter Conference on Applications of Computer Vision Workshops, Waikola, USA, 2021: 117–124.
[16]	QI C R, YI Li, SU Hao, et al. Pointnet++: Deep hierarchical feature learning on point sets in a metric space[C]. The 31st International Conference on Neural Information Processing Systems, Long Beach, USA, 2017: 5105–5114.
[17]	PENG Haotian, ZHOU Bin, YIN Liyuan, et al. Semantic part segmentation of single-view point cloud[J]. Science China Information Sciences, 2020, 63(12): 224101. doi: 10.1007/s11432-018-9689-9
[18]	ZHANG Jinming, HU Xiangyun, and DAI Hengming. A graph-voxel joint convolution neural network for ALS point cloud segmentation[J]. IEEE Access, 2020, 8: 139781–139791. doi: 10.1109/ACCESS.2020.3013293
[19]	SU Hang, JAMPANI V, SUN Deqing, et al. SPLATNet: Sparse lattice networks for point cloud processing[C]. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, USA, 2018: 2530–2539.
[20]	秦彩杰, 管强. 三维点云数据分割研究现状[J]. 宜宾学院学报, 2017, 17(6): 30–35. doi: 10.3969/j.issn.1671-5365.2017.06.007 QIN Caijie and GUAN Qiang. Research status of 3D point cloud data segmentation[J]. Journal of Yibin University, 2017, 17(6): 30–35. doi: 10.3969/j.issn.1671-5365.2017.06.007
[21]	CUI Yaodong, CHEN Ren, CHU Wenbo, et al. Deep learning for image and point cloud fusion in autonomous driving: A review[J]. IEEE Transactions on Intelligent Transportation Systems, 2022, 23(2): 722–739.
[22]	ZHANG Jiaying, ZHAO Xiaoli, CHEN Zheng, et al. A review of deep learning-based semantic segmentation for point cloud[J]. IEEE Access, 2019, 7: 179118–179133. doi: 10.1109/ACCESS.2019.2958671
[23]	俞斌, 董晨, 刘延华, 等. 基于深度学习的点云分割方法综述[J]. 计算机工程与应用, 2020, 56(1): 38–45. doi: 10.3778/j.issn.1002-8331.1910-0157 YU Bin, DONG Chen, LIU Yanhua, et al. Deep learning based point cloud segmentation: A survey[J]. Computer Engineering and Applications, 2020, 56(1): 38–45. doi: 10.3778/j.issn.1002-8331.1910-0157
[24]	LIU Kui and KANG Guixia. Multiview convolutional neural networks for lung nodule classification[J]. The Journal of Engineering, 2017, 27(1): 12–22. doi: 10.1002/ima.22206
[25]	SU Hang, MAJI S, KALOGERAKIS E, et al. Multi-view convolutional neural networks for 3D shape recognition[C]. 2015 IEEE International Conference on Computer Vision, Santiago, Chile, 2015: 945–953.
[26]	BOULCH A, GUERRY J, LE SAUX B, et al. SnapNet: 3D point cloud semantic labeling with 2D deep segmentation networks[J]. Computers & Graphics, 2018, 71: 189–198. doi: 10.1016/j.cag.2017.11.010
[27]	YOU Haoxuan, FENG Yifan, ZHAO Xibin, et al. PVRNet: Point-view relation neural network for 3D shape recognition[C]. The 33rd AAAI Conference on Artificial Intelligence, Honolulu, USA, 2019: 9119–9126.
[28]	WANG Wei, XU Yuan, REN Yingchao, et al. Parsing of urban facades from 3D point clouds based on a novel multi-view domain[J]. Photogrammetric Engineering & Remote Sensing, 2021, 87(4): 283–293. doi: 10.14358/PERS.87.4.283
[29]	郑阳, 林春雨, 廖康, 等. 场景视点偏移的激光雷达点云分割[J]. 中国图象图形学报, 2021, 26(10): 2514–2523. doi: 10.11834/jig.200424 ZHENG Yang, LIN Chunyu, LIAO Kang, et al. LiDAR point cloud segmentation through scene viewpoint offset[J]. Journal of Image and Graphics, 2021, 26(10): 2514–2523. doi: 10.11834/jig.200424
[30]	KUNDU A, YIN Xiaoqi, FATHI A, et al. Virtual multi-view fusion for 3D semantic segmentation[C]. The 16th European Conference on Computer Vision, Glasgow, UK, 2020: 518–535.
[31]	WU Zhirong, SONG Shuran, KHOSLA A, et al. 3D shapenets: A deep representation for volumetric shapes[C]. 2015 IEEE Conference on Computer Vision and Pattern Recognition, Boston, USA, 2015: 1912–1920.
[32]	HUANG Jing and YOU Suya. Point cloud labeling using 3D convolutional neural network[C]. The 2016 23rd International Conference on Pattern Recognition, Cancun, Mexico, 2016: 2670–2675.
[33]	DAI A and NIEßNER M. 3DMV: Joint 3D-multi-view prediction for 3D semantic scene segmentation[C]. The 15th European Conference on Computer Vision, Munich, Germany, 2018: 458–474.
[34]	LE T and DUAN Ye. PointGrid: A deep network for 3D shape understanding[C]. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, USA, 2018: 9204–9214.
[35]	KUMAWAT S and RAMAN S. LP-3DCNN: Unveiling local phase in 3D convolutional neural networks[C]. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, USA, 2019: 4898–4907.
[36]	TANG Haotian, LIU Zhijian, ZHAO Shengyu, et al. Searching efficient 3D architectures with sparse point-voxel convolution[C]. The 16th European Conference on Computer Vision, Glasgow, UK, 2020: 685–702.
[37]	孙一珺, 胡辉, 李子钥, 等. 适用于点云数据的注意力机制研究[J/OL]. 计算机工程与应用. https://kns.cnki.net/kcms/detail/detail.aspx?dbcode=CAPJ&dbname=CAPJLAST&filename=JSGG20211011003&uniplatform=NZKPT&v=3Qinehu7XHJEb8HcZQOml03a3GuI8TbdofNxjvnLvZ_l8jGtSqM5aNjUQVMqm-LC, 2021. SUN Yijun, HU Hui, LI Ziyue, et al. Research on attention mechanism for point cloud data[J/OL]. Computer Engineering and Applications. https://kns.cnki.net/kcms/detail/detail.aspx?dbcode=CAPJ&dbname=CAPJLAST&filename=JSGG20211011003&uniplatform=NZKPT&v=3Qinehu7XHJEb8HcZQOml03a3GuI8TbdofNxjvnLvZ_l8jGtSqM5aNjUQVMqm-LC, 2021.
[38]	XIE Saining, LIU Sainan, CHEN Zeyu, et al. Attentional ShapeContextNet for point cloud recognition[C]. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, USA, 2018: 4606–4615.
[39]	ZHANG Zhiyuan, HUA B S, CHEN Wei, et al. Global context aware convolutions for 3D point cloud understanding[C]. 2020 International Conference on 3D Vision, Fukuoka, Japan, 2020: 210–219.
[40]	XU Mutian, DING Runyu, ZHAO Hengshuang, et al. PAConv: Position adaptive convolution with dynamic kernel assembling on point clouds[C]. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, USA, 2021: 3172–3181.
[41]	LI Guohao, QIAN Guocheng, DELGADILLO I C, et al. SGAS: Sequential greedy architecture search[C]. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, USA, 2020: 1617–1627.
[42]	XU Yifan, FAN Tianqi, XU Mingye, et al. SpiderCNN: Deep learning on point sets with parameterized convolutional filters[C]. The 15th European Conference on Computer Vision, Munich, Germany, 2018: 95–105.
[43]	HUA B S, TRAN M K, and YEUNG S K. Pointwise convolutional neural networks[C]. Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, USA, 2018: 984–993.
[44]	KOMARICHEV A, ZHONG Zichun, and HUA Jing. A-CNN: Annularly convolutional neural networks on point clouds[C]. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, USA, 2019: 7413–7422.
[45]	高金金, 李潞洋. 基于局部关系卷积的点云分类与分割模型[J]. 计算机工程与应用, 2022, 58(19): 276–283. GAO Jinjin and LI Luyang. Local relation convolution network for 3D point cloud classification and segmentation[J]. Computer Engineering and Applications, 2022, 58(19): 276–283.
[46]	LIU Yongcheng, FAN Bin, XIANG Shiming, et al. Relation-shape convolutional neural network for point cloud analysis[C]. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, USA, 2019: 8887–8896.
[47]	LIU Yongcheng, FAN Bin, MENG Gaofeng, et al. Densepoint: Learning densely contextual representation for efficient point cloud processing[C]. 2019 IEEE/CVF International Conference on Computer Vision, Seoul, Korea (South), 2019: 5238–5247.
[48]	WU Wenxuan, QI Zhongang, and LI Fuxin. PointConv: Deep convolutional networks on 3D point clouds[C]. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, USA, 2019: 9613–9622,
[49]	LIN Zhihao, YU Shengyu, WANG Y C F. Convolution in the cloud: Learning deformable kernels in 3D graph convolution networks for point cloud analysis[C]. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, USA, 2020: 1797–1806.
[50]	WANG Fei, ZHANG Xing, JIANG Yong, et al. PatchCNN: An explicit convolution operator for point clouds perception[J]. IEEE Geoscience and Remote Sensing Letters, 2021, 18(4): 726–730. doi: 10.1109/LGRS.2020.2981507
[51]	SHEN Yiru, FENG Chen, YANG Yaoqing, et al. Mining point cloud local structures by kernel correlation and graph pooling[C]. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, USA, 2018: 4548–4557.
[52]	WANG Lei, HUANG Yuchun, HOU Yaolin, et al. Graph attention convolution for point cloud semantic segmentation[C]. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, USA, 2019: 10288–10297.
[53]	FENG Yifan, YOU Haoxuan, ZHANG Zizhao, et al. Hypergraph neural networks[C]. The 33rd AAAI Conference on Artificial Intelligence, Hawaii, USA, 2019: 3558–3565.
[54]	JIANG Jianwen, WEI Yuxuan, FENG Yifan, et al. Dynamic hypergraph neural networks[C]. The 28th International Joint Conference on Artificial Intelligence, Macao, China, 2019: 2635–2641.
[55]	XU Qiangeng, SUN Xudong, WU C Y, et al. Grid-GCN for fast and scalable point cloud learning[C]. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, USA, 2020: 5660–5669.
[56]	ENGELMANN F, KONTOGIANNI T, HERMANS A, et al. Exploring spatial context for 3D semantic segmentation of point clouds[C]. 2017 IEEE International Conference on Computer Vision Workshops, Venice, Italy, 2017: 716–724.
[57]	YE Xiaoqing, LI Jiamao, HUANG Hexiao, et al. 3D recurrent neural networks with context fusion for point cloud semantic segmentation[C]. Proceedings of the 15th European Conference on Computer Vision, Munich, Germany, 2018: 415–430.
[58]	ZHAO Zongyue, LIU Min, and RAMANI K. DAR-Net: Dynamic aggregation network for semantic scene segmentation[J]. arXiv: 1907.12022.
[59]	LIU Fangyu, LI Shuaipeng, ZHANG Liqiang, et al. 3DCNN-DQN-RNN: A deep reinforcement learning framework for semantic parsing of large-scale 3D point clouds[C]. 2017 IEEE International Conference on Computer Vision, Venice, Italy, 2017: 5679–5688.
[60]	QIN Can, YOU Haoxuan, WANG Lichen, et al. PointDAN: A multi-scale 3D domain adaption network for point cloud representation[C]. The 33rd International Conference on Neural Information Processing Systems, Vancouver, Canada, 2019: 7192–7203.
[61]	HAN Zhizhong, WANG Xiyang, LIU Yushen, et al. Multi-angle point cloud-VAE: Unsupervised feature learning for 3D point clouds from multiple angles by joint self-reconstruction and half-to-half prediction[C]. 2019 IEEE/CVF International Conference on Computer Vision, Seoul, Korea (South), 2019: 10441–10450.
[62]	CHEN Nenglun, LIU Lingjie, CUI Zhiming, et al. Unsupervised learning of intrinsic structural representation points[C]. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, USA, 2020: 9118–9127.
[63]	ZAMORSKI M, ZIEBA M, KLUKOWSKI P, et al. Adversarial autoencoders for compact representations of 3D point clouds[J]. Computer Vision and Image Understanding, 2020, 193: 102921. doi: 10.1016/j.cviu.2020.102921
[64]	RAO Yongming, LU Jiwen, and ZHOU Jie. Global-local bidirectional reasoning for unsupervised representation learning of 3D point clouds[C]. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, USA, 2020: 5375–5384.
[65]	CHANG A X, FUNKHOUSER T, GUIBAS L, et al. ShapeNet: An information-rich 3D model repository[J]. arXiv: 1512.03012, 2015.
[66]	DAI A, CHANG A X, SAVVA M, et al. ScanNet: Richly-annotated 3D reconstructions of indoor scenes[C]. 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, USA, 2017: 2432–2443.
[67]	HACKEL T, SAVINOV N, LADICKY L, et al. Semantic3D. net: A new large-scale point cloud classification benchmark[J]. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2017, IV-1/W1: 91–98. doi: 10.5194/isprs-annals-IV-1-W1-91-2017