面向三维高效视频编码的深度图错误隐藏

周洋; 吴佳忆; 陆宇; 殷海兵

doi:10.11999/JEIT180926

面向三维高效视频编码的深度图错误隐藏

doi: 10.11999/JEIT180926

杭州电子科技大学通信工程学院杭州 310018

基金项目: 浙江省自然科学基金(LY17F020027)，国家自然科学基金(61401132, 61572449)

详细信息

作者简介:
周洋：男，1979年生，副教授，硕士生导师，研究方向为视频分析和三维视频编解码

吴佳忆：男，1994年生，硕士生，研究方向为视频差错控制技术

陆宇：男，1977年生，讲师，研究方向为视频压缩技术

殷海兵：男，1974年生，教授，研究方向为视频编码与压缩芯片设计

通讯作者:
周洋　zhouyang_hz@126.com

中图分类号: TP391
计量
- 文章访问数: 2118
- HTML全文浏览量: 966
- PDF下载量: 41
- 被引次数: 0
出版历程
- 收稿日期: 2018-09-30
- 修回日期: 2019-04-05
- 网络出版日期: 2019-05-21
- 刊出日期: 2019-11-01

Depth Map Error Concealment for 3D High Efficiency Video Coding

Institute of Telecommunication Engineering, Hangzhou Dianzi University, Hangzhou 310018, China

Funds: Zhejiang Province Natural Science Fundation (LY17F020027), The National Natural Science Foundation of China (61401132, 61572449)

摘要

摘要: 基于多视点视频序列视点内、视点间存在的相关性，并结合视点间运动矢量共享技术，该文提出一种面向3维高效视频编码中深度序列传输丢包的错误隐藏算法。首先，根据3D高效视频编码(3D-HEVC)的分层B帧预测(HBP)结构和深度图纹理特征，将深度图丢失块分成运动块和静止块；然后，对于受损运动块，使用结合纹理结构的外边界匹配准则来选择相对最优的运动/视差矢量进行基于位移矢量补偿的错误掩盖，而对受损静止块采用参考帧直接拷贝进行快速错误隐藏；最后，使用参考帧拆分重组来获取新的运动/视差补偿块对修复质量较差的重建块进行质量提升。实验结果表明：相较于近年提出的对比算法，该文算法隐藏后的深度帧平均峰值信噪比(PSNR)能提升0.25～2.03 dB，结构相似度测量值(SSIM)能提升0.001～0.006，且修复区域的主观视觉质量与原始深度图更接近。
- 3维高效视频编码 /
- 多视视频加深度 /
- 匹配准则 /
- 视差矢量 /
- 错误隐藏
Abstract: By using the intra-view and inter-view correlations and the motion vector-sharing, a depth map error concealment approach is proposed for 3D video coding based on the High Efficiency Video Coding (3D-HEVC) to combat the packet loss of the depth video transmission. Based on the Hierarchical B-frame Prediction (HBP) structure in 3D-HEVC and textured features of the depth map, all the lost coding units are firstly categorized into two classes, i.e., motion blocks and static blocks. Then, according to the outer boundary matching criterion combining the texture structure, the optimal motion/disparity vector is chosen for the damaged motion blocks to conduct the motion/disparity compensation based error concealment. Whereas, the direct copy is applied to concealling the damaged static blocks quickly. Finally, for the concealed blocks whose qualities are not ideal, the new motion/disparity compensation blocks reconstructed by the reference frames recombination are applied to improning the qualities of those blocks. The experimental results show that the repaired depth map concealed by the proposed approach can achieve 0.25～2.03 dB gain in term of the Peak-Signal-to-Noise Ratio (PSNR) and 0.001～0.006 gain in term of Structural Similarity Index Measure(SSIM). Moreover, the subjective visual quality of the repaired area is better in lines with the original depth maps.
- 3D-High Efficiency Video Coding(3D-HEVC) /
- Multi-view video plus depth /
- Matching criterion /
- Disparity vector /
- Error concealment

HTML全文

图 1 算法总体流程图

下载: 全尺寸图片幻灯片

图 2 丢失块周围邻块位移矢量的选择

下载: 全尺寸图片幻灯片

图 3 PSNR逐帧测试结果曲线图

下载: 全尺寸图片幻灯片

图 4 Poznanstreet第3视点第3帧错误隐藏结果

下载: 全尺寸图片幻灯片

图 5 Kendo第3视点第3帧主观测试结果

下载: 全尺寸图片幻灯片

表 1 组合参考块对应表

参考列表	参考帧号	L0				L1
参考列表	参考帧号	R0₀	R0₁	…	R0_K	R1₀	R1₁	…	R1_K
L0	R0₀	P₀	–	–		B₀₀	B₀₁	B₀_i	B₀_K
	R0₁	–	P₁	–		B₁₀	B₁₁	B₁_i	B₁_K
	∶	–	–	P_i		B_i₀	B_i₁	B_ii	B_iK
	R0_K				P_K	B_K₀	B_K₁	B_Ki	B_KK
L1	R1₀	B₀₀	B₁₀	B_i₀	B_K₀	P_K₊₁	–	–
	R1₁	B₀₁	B₁₁	B_i₁	B_K₁	–	P_K₊₂	–
	∶	B₀_i	B₁_i	B_ii	B_Ki	–	–	P_K₊_i
	R1_K	B₀_K	B₁_K	B_iK	B_KK				P₂_K

下载: 导出CSV

表 2 QP=30, 25时各测试序列重建图像平均PSNR计算结果(dB)

QP	序列	原始	错误率50%				错误率25%				错误率10%
QP	序列	原始	文献[6]	文献[7]	MVC	本文算法	文献[6]	文献[7]	MVC	本文算法	文献[6]	文献[7]	MVC	本文算法
30	Kendo	42.761	36.406	36.122	35.894	37.103	37.896	37.359	36.786	38.310	39.449	38.925	38.025	39.833
	Bookarrival	39.923	36.932	35.911	35.656	37.525	37.865	37.171	36.526	38.286	38.338	37.699	37.389	38.747
	Poznanstreet	44.028	39.593	38.834	36.662	40.864	41.155	41.028	40.799	41.890	41.450	41.219	40.983	42.171
	Balloon	40.574	38.713	38.512	37.691	39.269	39.514	39.456	38.347	39.839	39.706	39.638	38.485	39.958
25	Kendo	45.874	36.736	36.548	36.096	37.499	39.227	38.058	37.287	39.655	41.368	39.688	39.133	41.501
	Bookarrival	41.625	37.981	36.733	36.354	38.270	38.411	37.488	36.947	39.921	39.728	39.196	38.463	40.187
	Poznanstreet	46.122	39.955	38.928	37.066	41.561	41.723	41.178	40.802	42.319	42.113	41.797	41.207	42.622
	Balloon	44.022	40.829	39.434	39.156	41.247	41.899	41.277	40.894	42.149	42.234	42.158	41.084	42.612

下载: 导出CSV

表 3 QP=30, 25时各测试序列重建图像平均SSIM计算结果

QP	序列	原始	错误率50%				错误率25%				错误率10%
QP	序列	原始	文献[6]	文献[7]	MVC	本文算法	文献[6]	文献[7]	MVC	本文算法	文献[6]	文献[7]	MVC	本文算法
30	Kendo	0.9867	0.9708	0.9688	0.9647	0.9721	0.9780	0.9776	0.9733	0.9785	0.9819	0.9816	0.9766	0.9822
	Bookarrival	0.9565	0.9482	0.9457	0.9422	0.9495	0.9503	0.9481	0.9452	0.9510	0.9525	0.9518	0.9477	0.9534
	Poznanstreet	0.9795	0.9732	0.9720	0.9714	0.9759	0.9746	0.9740	0.9738	0.9767	0.9751	0.9750	0.9744	0.9769
	Balloon	0.9767	0.9738	0.9733	0.9720	0.9743	0.9745	0.9742	0.9736	0.9751	0.9747	0.9746	0.9743	0.9757
25	Kendo	0.9874	0.9722	0.9696	0.9660	0.9751	0.9789	0.9783	0.9765	0.9796	0.9828	0.9824	0.9795	0.9832
	Bookarrival	0.9701	0.9577	0.9547	0.9539	0.9598	0.9634	0.9627	0.9613	0.9641	0.9643	0.9642	0.9633	0.9655
	Poznanstreet	0.9878	0.9818	0.9803	0.9796	0.9824	0.9825	0.9821	0.9816	0.9848	0.9843	0.9841	0.9824	0.9849
	Balloon	0.9855	0.9810	0.9803	0.9796	0.9821	0.9834	0.9833	0.9828	0.9839	0.9841	0.9836	0.9835	0.9844

下载: 导出CSV

表 4 各测试序列解码运行时间(s)

序列	原解码	文献[6]	文献[7]	MVC	本文
Kendo	188.71	231.45	198.83	192.07	194.19
Balloon	172.35	216.14	178.13	173.82	174.69
Bookarrival	175.86	219.35	183.19	177.18	179.13
Poznanstreet	483.67	558.15	498.68	484.68	486.07

下载: 导出CSV

参考文献(20)

LI Yufeng and CHEN Ruining. Motion vector recovery for video error concealment based on plane fitting[J]. Computer Engineering &Applications, 2018, 54(2): 182–187. doi: 10.3778/j.issn.1002-8331.1608-0085

XU Dawen and WANG Rangding. Two-dimensional reversible data hiding-based approach for intra-frame error concealment in H.264/AVC[J]. Journal of Visual Communication& Image Representation, 2016, 47(2): 369–379. doi: 10.1016/j.jvcir.2013.12.008

ZABIHI S M, GHANEI Y H, and MEHRSHAD N. Adaptive temporal error concealment method based on the MB behavior estimation in the video[C]. The 7th International Conference on Computer and Knowledge Engineering. Mashhad, Iran, 2017: 193–198.

EL-SHAFAI W, EL-RABAIE S, EL-HALAWANY M M, et al. Recursive Bayesian filtering-based error concealment scheme for 3D video communication over severely lossy wireless channels[J]. Circuits Systems & Signal Processing, 2018(20): 1–32. doi: 10.1007/s00034-018-0786-8

周洋, 蒋刚毅, 郁梅, 等. 面向HBP编码格式的立体视频B帧整帧丢失分层错误隐藏算法[J]. 电子与信息学报, 2014, 36(2): 377–383. doi: 10.3724/SP.J.1146.2013.00528

ZHOU Yang, JIANG Gangyi, YU Mei, et al. A hierarchical error concealment algorithm for entire B-frame loss in stereoscopic video coding with hierarchical B pictures[J]. Journal of Electronics &Information Technology, 2014, 36(2): 377–383. doi: 10.3724/SP.J.1146.2013.00528

SONG K, CHUNG T, OH Y, et al. Error concealment of multi-view video sequences using inter-view and intra-view correlations[J]. Journal of Visual Communication & Image Representation, 2009, 20(4): 281–292. doi: 10.1016/j.jvcir.2009.02.002

PENG Yantsung and COSMAN P C. Weighted boundary matching error concealment for HEVC using block partition decisions[C]. The 48th Asilomar Conference on Signals, Systems & Computers, Grove, USA, 2014: 921–925. doi: 10.1109/ACSSC.2014.7094587.

KHATTAK S, MAUGEY T, HAMZAOUI R, et al. Temporal and inter-view consistent error concealment technique for multiview plus depth video[J]. IEEE Transactions on Circuits & Systems for Video Technology, 2016, 26(5): 829–840. doi: 10.1109/TCSVT.2015.2418631

LIU Shujie, CHEN Ying, WANG Yekui, et al. Frame loss error concealment for multiview video coding[C]. IEEE International Symposium on Circuits and Systems (ISCAS 2008), Washington, USA, 2008: 3470–3473.

ZHOU Yuan, XIANG Wei, and WANG Gengkun. Frame loss concealment for multiview video transmission over wireless multimedia sensor networks[J]. IEEE Sensors Journal, 2015, 15(3): 1892–1901. doi: 10.1109/JSEN.2014.2366511

HUANG Pincheng, LIN Jieru, LI Gwolong, et al. Improved depth-assisted error concealment algorithm for 3D video transmission[J]. IEEE Transactions on Multimedia, 2017, 19(11): 2625–2632. doi: 10.1109/TMM.2017.2694218

TAI Shenchuan, WANG Chuenching, HONG Chienshiang, et al. An efficient full frame algorithm for object-based error concealment in 3D depth-based video[J]. Multimedia Tools & Applications, 2016, 75(16): 9927–9947. doi: 10.1007/s11042-015-2899-4

时琳, 刘荣科, 李君烨. 基于深度信息的立体视频错误隐藏方法[J]. 电子与信息学报, 2012, 34(7): 1678–1684. doi: 10.3724/SP.J.1146.2011.01107

SHI Ling, LIU Rongke, and LI Junye. Error concealment based on depth information for stereoscopic video coding[J]. Journal of Electronics &Information Technology, 2012, 34(7): 1678–1684. doi: 10.3724/SP.J.1146.2011.01107

YANG Meng, LAN Xuguang, ZHENG Nanning, et al. Depth-assisted temporal error concealment for intra frame slices in 3-D video[J]. IEEE Transactions on Broadcasting, 2014, 60(2): 385–393. doi: 10.1109/TBC.2014.2321676

YAN Bo and ZHOU Jie. Efficient frame concealment for depth image-based 3-D video transmission[J]. IEEE Transactions on Multimedia, 2012, 14(3): 936–941. doi: 10.1109/TMM.2012.2184743

MERKLE P, MULLER K, MARPE D, et al. Depth intra coding for 3D video based on geometric primitives[J]. IEEE Transactions on Circuits & Systems for Video Technology, 2016, 26(3): 570–582. doi: 10.1109/TCSVT.2015.2407791

HEWAGE C T E R, WORRALL S, DOGAN S, et al. Frame concealment algorithm for stereoscopic video using motion vector sharing[C]. 2008 IEEE International Conference on Multimedia and Expo, Hannover, Germany, 2008: 485–488. doi: 10.1109/ICME.2008.4607477.

ASSUNCAO P, MARCELINO S, SOARES S, et al. Spatial error concealment for intra-coded depth maps in multiview video-plus-depth[J]. Multimedia Tools &Applications, 2017, 76(12): 13835–13858. doi: 10.1007/s11042-016-3766-7

HTM Reference Software 14.0. http://hevc.kw.bbc.co.uk/git/w/jctvc-3de.git, 2016.

WANG Zhou, BOVIK A C, SHEIKH H R, et al. Image quality assessment: From error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13(4): 600–612. doi: 10.3969/j.issn.1006-8961.2006.12.00.003

施引文献

资源附件(0)

访问统计