A Review of Image Reconstruction Based on Event Cameras

XU Qi; DENG Jie; SHEN Jiangrong; TANG Huajin; PAN Gang

doi:10.11999/JEIT221456

Volume 45 Issue 8

Aug. 2023

Turn off MathJax

Article Contents

Article Navigation > Journal of Electronics & Information Technology > 2023 > 45(8): 2699-2709

XU Qi, DENG Jie, SHEN Jiangrong, TANG Huajin, PAN Gang. A Review of Image Reconstruction Based on Event Cameras[J]. Journal of Electronics & Information Technology, 2023, 45(8): 2699-2709. doi: 10.11999/JEIT221456

Citation:

XU Qi, DENG Jie, SHEN Jiangrong, TANG Huajin, PAN Gang. A Review of Image Reconstruction Based on Event Cameras[J]. Journal of Electronics & Information Technology, 2023, 45(8): 2699-2709. doi: 10.11999/JEIT221456

Citation:

PDF( 1761 KB)

A Review of Image Reconstruction Based on Event Cameras

doi: 10.11999/JEIT221456

1.
School of Artificial Intelligence, Dalian University of Technology, Dalian 116024, China
2.
College of Computer Science and Technology, Zhejiang University, Hangzhou 310027, China

Funds: The National Natural Science Foundation of China (62206037), The National Key R&D Program of China (2021ZD0109803), The Open Research Fund from Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ) (GML-KF-22-11), The Fundamental Research Funds for the Central Universities (DUT21RC(3)091)

Received Date: 2022-11-21
Rev Recd Date: 2023-06-06

Available Online: 2023-06-19

Publish Date: 2023-08-21

Abstract

Abstract

Event cameras are bio-inspired sensors that outputs a stream of events when the brightness change of pixels exceeds the threshold. This type of visual sensor asynchronously outputs events that encode the time, location and sign of the brightness changes. Hence, event cameras offer attractive properties, such as high temporal resolution, very high dynamic range, low latency, low power consumption, and high pixel bandwidth. It can capture information in high-speed motion and high-dynamic scenes, which can be used to reconstruct high-dynamic range and high-speed motion scenes. Brightness images obtained by image reconstruction can be interpreted as a representation, and be used for recognition, segmentation, tracking and optical flow estimation, which is one of the important research directions in the field of vision. This survey first briefly introduces event cameras from their working principle, developmental history, advantages, and challenges of event cameras. Then, the working principles of various types of event cameras and some event camera-based image reconstruction algorithms are introduced. Finally, the challenges and future trends faced by event cameras are described, and the article is concluded.
- Event camera,
- Dynamic vision sensors,
- Image reconstruction

FullText(HTML)

References(40)

References

[1]	KIM H, LEUTENEGGER S, and DAVISON A J. Real-time 3D reconstruction and 6-DoF tracking with an event camera[C]. The 14th European Conference on Computer Vision, Amsterdam, The Netherlands, 2016: 349–364.
[2]	VIDAL A R, REBECQ H, HORSTSCHAEFER T, et al. Ultimate SLAM? Combining events, images, and IMU for robust visual SLAM in HDR and high-speed scenarios[J]. IEEE Robotics and Automation Letters, 2018, 3(2): 994–1001. doi: 10.1109/LRA.2018.2793357
[3]	LAGORCE X, MEYER C, IENG S H, et al. Asynchronous event-based multikernel algorithm for high-speed visual features tracking[J]. IEEE Transactions on Neural Networks and Learning Systems, 2015, 26(8): 1710–1720. doi: 10.1109/TNNLS.2014.2352401
[4]	BARDOW P, DAVISON A J, and LEUTENEGGER S. Simultaneous optical flow and intensity estimation from an event camera[C]. 2016 IEEE Conference on Computer Vision and pattern Recognition, Las Vegas, USA, 2016: 884–892.
[5]	LAGORCE X, ORCHARD G, GALLUPPI F, et al. HOTS: A hierarchy of event-based time-surfaces for pattern recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(7): 1346–1359. doi: 10.1109/TPAMI.2016.2574707
[6]	SIRONI A, BRAMBILLA M, BOURDIS N, et al. HATS: Histograms of averaged time surfaces for robust event-based object classification[C]. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, USA, 2018: 1731–1740.
[7]	张远辉, 许璐钧, 徐柏锐, 等. 基于事件相机的人体动作识别方法[J]. 计量学报, 2022, 43(5): 583–589. doi: 10.3969/j.issn.1000-1158.2022.05.04 ZHANG Yuanhui, XU Lujun, XU Borui, et al. Human action recognition method based on event camera[J]. Acta Metrologica Sinica, 2022, 43(5): 583–589. doi: 10.3969/j.issn.1000-1158.2022.05.04
[8]	GALLEGO G, LUND J E A, MUEGGLER E, et al. Event-based, 6-DOF camera tracking from photometric depth maps[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(10): 2402–2412. doi: 10.1109/TPAMI.2017.2769655
[9]	GEHRIG D, LOQUERCIO A, DERPANIS K, et al. End-to-end learning of representations for asynchronous event-based data[C]. 2019 IEEE/CVF International Conference on Computer Vision, Seoul, Korea (South), 2019: 5633–5643.
[10]	CORDONE L, MIRAMOND B, and THIERION P. Object detection with spiking neural networks on automotive event data[C]. International Joint Conference on Neural Networks, Padua, Italy, 2022.doi: 10.1109/IJCNN55064.2022.9892618.
[11]	MUNDA G, REINBACHER C, and POCK T. Real-time intensity-image reconstruction for event cameras using manifold regularisation[J]. International Journal of Computer Vision, 2018, 126(12): 1381–1393. doi: 10.1007/s11263-018-1106-2
[12]	SCHEERLINCK C, BARNES N, and MAHONY R. Continuous-time intensity estimation using event cameras[C]. The 14th Asian Conference on Computer Vision, Perth, Australia, 2018: 308–324.
[13]	MAHOWALD M. VLSI analogs of neuronal visual processing: A synthesis of form and function[D]. [Ph. D. dissertation], California Institute of Technology, 1992.
[14]	LICHTSTEINER P, POSCH C, and DELBRUCK T. A 128×128 120 dB 15 µs latency asynchronous temporal contrast vision sensor[J]. IEEE Journal of Solid-State Circuits, 2008, 43(2): 566–576. doi: 10.1109/JSSC.2007.914337
[15]	POSCH C, MATOLIN D, and WOHLGENANNT R. A QVGA 143 dB dynamic range frame-free PWM image sensor with lossless pixel-level video compression and time-domain CDS[J]. IEEE Journal of Solid-State Circuits, 2011, 46(1): 259–275. doi: 10.1109/JSSC.2010.2085952
[16]	BRANDLI C, BERNER R, YANG Minhao, et al. A 240×180 130 db 3 µs latency global shutter spatiotemporal vision sensor[J]. IEEE Journal of Solid-State Circuits, 2014, 49(10): 2333–2341. doi: 10.1109/JSSC.2014.2342715
[17]	MOEYS D P, CORRADI F, LI Chenghan, et al. A sensitive dynamic and active pixel vision sensor for color or neural imaging applications[J]. IEEE Transactions on Biomedical Circuits and Systems, 2018, 12(1): 123–136. doi: 10.1109/TBCAS.2017.2759783
[18]	HUANG Jing, GUO Menghan, and CHEN Shoushun. A dynamic vision sensor with direct logarithmic output and full-frame picture-on-demand[C]. 2017 IEEE International Symposium on Circuits and Systems (ISCAS), Baltimore, USA, 2017: 1–4.
[19]	HUANG Tiejun, ZHENG Yajing, YU Zhaofei, et al. 1000×faster Camera and machine vision with ordinary devices[EB/OL]. https://www.sciecedirect.com/scince/articel/pii/s2095809922002077.
[20]	FOSSUM E R. CMOS image sensors: Electronic camera-on-a-chip[J]. IEEE Transactions on Electron Devices, 1997, 44(10): 1689–1698. doi: 10.1109/16.628824
[21]	DONG Siwei, HUANG Tiejun, and TIAN Yonghong. Spike camera and its coding methods[C]. 2017 Data Compression Conference, Snowbird, USA, 2021: 437.
[22]	COOK M, GUGELMANN L, JUG F, et al. Interacting maps for fast visual interpretation[C]. 2011 International Joint Conference on Neural Networks, San Jose, USA, 2011: 770–776.
[23]	KIM H, HANDA A, BENOSMAN R, et al. Simultaneous mosaicing and tracking with an event camera[J]. J Solid State Circ, 2008, 43: 566–576.
[24]	BELBACHIR A N, SCHRAML S, MAYERHOFER M, et al. A novel HDR depth camera for real-time 3D 360° panoramic vision[C]. 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops, Columbus, USA, 2014: 425–432.
[25]	BARUA S, MIYATANI Y, and VEERARAGHAVAN A. Direct face detection and video reconstruction from event cameras[C]. 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Placid, USA, 2016: 1–9.
[26]	REBECQ H, RANFTL R, KOLTUN V, et al. Events-to-video: Bringing modern computer vision to event cameras[C]. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, USA, 2019: 3852–3861.
[27]	REBECQ H, RANFTL R, KOLTUN V, et al. High speed and high dynamic range video with an event camera[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43(6): 1964–1980. doi: 10.1109/TPAMI.2019.2963386
[28]	SCHEERLINCK C, REBECQ H, GEHRIG D, et al. Fast image reconstruction with an event camera[C]. 2020 IEEE Winter Conference on Applications of Computer Vision, Snowmass, USA, 2020: 156–163.
[29]	REBECQ H, GEHRIG D, and SCARAMUZZA D. ESIM: An open event camera simulator[C]. The 2nd Conference on Robot Learning, Zürich, Switzerland, 2018: 969–982.
[30]	WANG Lin, MOSTAFAVI I S M, HO Y S, et al. Event-based high dynamic range image and very high frame rate video generation using conditional generative adversarial networks[C]. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, USA, 2019: 10073–10082.
[31]	WANG Lin, KIM T K, and YOON K J. EventSR: From asynchronous events to image reconstruction, restoration, and super-resolution via end-to-end adversarial learning[C]. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, USA, 2020: 8312–8322.
[32]	WANG Lin, KIM T K, and YOON K J. Joint framework for single image reconstruction and super-resolution with an event camera[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(11): 7657–7673. doi: 10.1109/TPAMI.2021.3113352
[33]	ZHU Lin, DONG Siwei, LI Jianing, et al. Retina-like visual image reconstruction via spiking neural model[C]. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, USA, 2020: 1435–1443.
[34]	ZHU Lin, WANG Xiao, CHANG Yi, et al. Event-based video reconstruction via potential-assisted spiking neural network[C]. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, USA, 2022: 3584–3594.
[35]	DUWEK H C, SHALUMOV A, and TSUR E E. Image reconstruction from neuromorphic event cameras using Laplacian-prediction and Poisson integration with spiking and artificial neural networks[C]. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Nashville, USA, 2021: 1333–1341.
[36]	BRANDLI C, MULLER L, and DELBRUCK T. Real-time, high-speed video decompression using a frame-and event-based DAVIS sensor[C]. 2014 IEEE International Symposium on Circuits and Systems (ISCAS), Melbourne, Australia, 2014: 686–689.
[37]	LIU Hanchao, ZHANG Fanglue, MARSHALL D, et al. High-speed video generation with an event camera[J]. The Visual Computer, 2017, 33(6): 749–759.
[38]	PAN Liyuan, SCHEERLINCK C, YU Xin, et al. Bringing a blurry frame alive at high frame-rate with an event camera[C]. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, USA, 2019: 6813–6822.
[39]	PAN Liyuan, HARTLEY R, SCHEERLINCK C, et al. High frame rate video reconstruction based on an event camera[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(5): 2519–2533. doi: 10.1109/TPAMI.2020.3036667
[40]	SHEDLIGERI P and MITRA K. Photorealistic image reconstruction from hybrid intensity and event-based sensor[J]. Journal of Electronic Imaging, 2019, 28(6): 063012. doi: 10.1117/1.JEI.28.6.063012