A Review of Neural Radiance Field Approaches for Scene Reconstruction of Satellite Remote Sensing Imagery

ZHOU Xin; WANG Yang; SUN Xian; LIN Daoyu; LIU Junyi; FU Kun

doi:10.11999/JEIT240202

Volume 46 Issue 5

May 2024

Turn off MathJax

Article Contents

Article Navigation > Journal of Electronics & Information Technology > 2024 > 46(5): 1582-1590

ZHOU Xin, WANG Yang, SUN Xian, LIN Daoyu, LIU Junyi, FU Kun. A Review of Neural Radiance Field Approaches for Scene Reconstruction of Satellite Remote Sensing Imagery[J]. Journal of Electronics & Information Technology, 2024, 46(5): 1582-1590. doi: 10.11999/JEIT240202

Citation:

ZHOU Xin, WANG Yang, SUN Xian, LIN Daoyu, LIU Junyi, FU Kun. A Review of Neural Radiance Field Approaches for Scene Reconstruction of Satellite Remote Sensing Imagery[J]. Journal of Electronics & Information Technology, 2024, 46(5): 1582-1590. doi: 10.11999/JEIT240202

Citation:

PDF( 7611 KB)

A Review of Neural Radiance Field Approaches for Scene Reconstruction of Satellite Remote Sensing Imagery

doi: 10.11999/JEIT240202

ZHOU Xin^{1, 2, 3, 4},
WANG Yang^{1, 2},
SUN Xian^{1, 2, 3},
LIN Daoyu^{1, 2},
LIU Junyi^{1, 2
,
,},
FU Kun^{1, 2, 3}

1.
Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100190, China
2.
Key Laboratory of Network Information System Technology(NIST), Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100190, China
3.
University of Chinese Academy of Sciences, Beijing 100190, China
4.
School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences, Beijing 100190, China

Received Date: 2024-03-25
Rev Recd Date: 2024-05-13

Available Online: 2024-05-17

Publish Date: 2024-05-30

Abstract

Abstract

High-resolution satellite remote sensing images have been recognized as an indispensable means for understanding geographical spaces, and their role in areas such as urban mapping, ecological monitoring, and navigation, has become increasingly important. The use of satellite remote sensing images for large-scale 3D reconstruction of the Earth’s surface is currently a subject of active research in the fields of computer vision and photogrammetry. Neural Radiance Fields (NeRF), which utilizes differentiable rendering to learn implicit representations of scenes, has achieved the most realistic visual effects in novel view synthesis tasks of complex scenes and has attracted significant attention in the field of 3D scene reconstruction and rendering. Recent research has been primarily focused on using neural radiance field technology to extract scene representation and reconstruction from satellite remote sensing images. Ray space optimization, scene representation optimization, and efficient model training are mainly focused on by the neural radiance field methods for satellite remote sensing images. The latest progress in the application of neural radiance field technology in satellite remote sensing is comprehensively summarized in this paper. First, the basic concepts of neural radiance field technology and related datasets are introduced. Then a classification framework of neural radiance field methods for satellite remote sensing images is proposed to systematically review and organize the research progress of this technology in the field of satellite remote sensing. The relevant results of the application of neural radiance field technology in actual satellite remote sensing scenarios are detailed. Finally, analysis and discussion are conducted based on the problems and challenges faced by current research, and future development trends and research directions are prospected.
- Neural Radiance Field(NeRF),
- Satellite remote sensing imagery,
- Neural rendering,
- Implicit representation

FullText(HTML)

References(31)

References

[1]	MILDENHALL B, SRINIVASAN P P, TANCIK M, et al. NeRF: Representing scenes as neural radiance fields for view synthesis[J]. Communications of the ACM, 2021, 65(1): 99–106. doi: 10.1145/3503250.
[2]	ZHANG Lulin and RUPNIK E. Sparsesat-NeRF: Dense depth supervised neural radiance fields for sparse satellite images[J]. arXiv preprint arXiv: 2309.00277, 2023. doi: 10.48550/arXiv.2309.00277.
[3]	BARRON J T, MILDENHALL B, TANCIK M, et al. Mip-NeRF: A multiscale representation for anti-aliasing neural radiance fields[C]. The 2021 IEEE/CVF International Conference on Computer Vision, Montreal, Canada, 2021: 5835–5844. doi: 10.1109/ICCV48922.2021.00580.
[4]	TANCIK M, SRINIVASAN P P, MILDENHALL B, et al. Fourier features let networks learn high frequency functions in low dimensional domains[C]. The 34th International Conference on Neural Information Processing Systems, Vancouver, Canada, 2020: 632.
[5]	TANCIK M, CASSER V, YAN Xinchen, et al. Block-NeRF: Scalable large scene neural view synthesis[C]. The 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, USA, 2022: 8238–8248. doi: 10.1109/CVPR52688.2022.00807.
[6]	MAX N. Optical models for direct volume rendering[J]. IEEE Transactions on Visualization and Computer Graphics, 1995, 1(2): 99–108. doi: 10.1109/2945.468400.
[7]	GAO K, GAO Yina, HE Hongjie, et al. NeRF: Neural radiance field in 3D vision, a comprehensive review[J]. arXiv preprint arXiv: 2210.00379, 2022. doi: 10.48550/arXiv.2210.00379.
[8]	LORENSEN W E and CLINE H E. Marching cubes: A high resolution 3D surface construction algorithm[M]. WOLFE R. Seminal Graphics: Pioneering Efforts that Shaped the Field. New York: Association for Computing Machinery, 1998: 347–353. doi: 10.1145/280811.281026.
[9]	MILDENHALL B, SRINIVASAN P P, ORTIZ-CAYON R, et al. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines[J]. ACM Transactions on Graphics, 2019, 38(4): 29. doi: 10.1145/3306346.3322980.
[10]	BARRON J T, MILDENHALL B, VERBIN D, et al. Mip-NeRF 360: Unbounded anti-aliased neural radiance fields[C]. The 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, USA, 2022: 5460–5469. doi: 10.1109/CVPR52688.2022.00539.
[11]	KNAPITSCH A, PARK J, ZHOU Qianyi, et al. Tanks and temples: Benchmarking large-scale scene reconstruction[J]. ACM Transactions on Graphics, 2017, 36(4): 78. doi: 10.1145/3072959.3073599.
[12]	BOSCH M, FOSTER K, CHRISTIE G, et al. Semantic stereo for incidental satellite images[C]. 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), Waikoloa, USA, 2019: 1524–1532. doi: 10.1109/WACV.2019.00167.
[13]	LE SAUX B, YOKOYA N, HANSCH R, et al. 2019 data fusion contest [technical committees][J]. IEEE Geoscience and Remote Sensing Magazine, 2019, 7(1): 103–105. doi: 10.1109/MGRS.2019.2893783.
[14]	MARÍ R, FACCIOLO G, and EHRET T. Sat-NeRF: Learning multi-view satellite photogrammetry with transient objects and shadow modeling using RPC cameras[C]. The 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, New Orleans, USA, 2022: 1310–1320. doi: 10.1109/CVPRW56347.2022.00137.
[15]	SCHÖNBERGER J L and FRAHM J M. Structure-from-motion revisited[C]. The 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, 2016: 4104–4113. doi: 10.1109/CVPR.2016.445.
[16]	DERKSEN D and IZZO D. Shadow neural radiance fields for multi-view satellite photogrammetry[C]. The 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Nashville, USA, 2021: 1152–1161. doi: 10.1109/CVPRW53098.2021.00126.
[17]	MARÍ R, FACCIOLO G, and EHRET T. Multi-date earth observation NeRF: The detail is in the shadows[C]. The 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Vancouver, Canada, 2023: 2035–2045. doi: 10.1109/CVPRW59228.2023.00197.
[18]	QU Yingjie and DENG Fei. Sat-mesh: Learning neural implicit surfaces for multi-view satellite reconstruction[J]. Remote Sensing, 2023, 15(17): 4297. doi: 10.3390/rs15174297.
[19]	MARTIN-BRUALLA R, RADWAN N, SAJJADI M S M, et al. Nerf in the wild: Neural radiance fields for unconstrained photo collections[C]. The 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, USA, 2021: 7206–7215. doi: 10.1109/CVPR46437.2021.00713.
[20]	SUN Cheng, SUN Min, and CHEN H T. Improved direct voxel grid optimization for radiance fields reconstruction[J]. arXiv preprint arXiv: 2206.05085, 2022. doi: 10.48550/arXiv.2206.05085.
[21]	CHEN Anpei, XU Zexiang, GEIGER A, et al. TensoRF: Tensorial radiance fields[C]. The 17th European Conference on Computer Vision, Tel Aviv, Israel, 2022: 333–350. doi: 10.1007/978-3-031-19824-3_20.
[22]	MÜLLER T, EVANS A, SCHIED C, et al. Instant neural graphics primitives with a multiresolution hash encoding[J]. ACM Transactions on Graphics, 2022, 41(4): 102. doi: 10.1145/3528223.3530127.
[23]	XIE Songlin, ZHANG Lei, JEON G, et al. Remote sensing neural radiance fields for multi-view satellite photogrammetry[J]. Remote Sensing, 2023, 15(15): 3808. doi: 10.3390/rs15153808.
[24]	ZHANG Tongtong and LI Yuanxiang. Fast satellite tensorial radiance field for multi-date satellite imagery of large size[J]. arXiv preprint arXiv: 2309.11767, 2023. doi: 10.48550/arXiv.2309.11767.
[25]	ROESSLE B, BARRON J T, MILDENHALL B, et al. Dense depth priors for neural radiance fields from sparse input views[C]. Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, USA, 2022: 12882–12891. doi: 10.1109/CVPR52688.2022.01255.
[26]	GABLEMAN M and KAK A. Incorporating season and solar specificity into renderings made by a NeRF architecture using satellite images[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024, 46(6): 4348–4365. doi: 10.1109/TPAMI.2024.3355069.
[27]	REMATAS K, LIU A, SRINIVASAN P, et al. Urban radiance fields[C]. The 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, USA, 2022: 12922–12932. doi: 10.1109/CVPR52688.2022.01259.
[28]	TURKI H, RAMANAN D, and SATYANARAYANAN M. Mega-NeRF: Scalable construction of large-scale NeRFs for virtual fly-throughs[C]. The 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, USA, 2022: 12912–12921. doi: 10.1109/CVPR52688.2022.01258.
[29]	HO J, JAIN A, and ABBEEL P. Denoising diffusion probabilistic models[C]. The 34th International Conference on Neural Information Processing Systems, Vancouver, Canada, 2020: 574.
[30]	RADFORD A, KIM J W, HALLACY C, et al. Learning transferable visual models from natural language supervision[C]. The 38th International Conference on Machine Learning, 2021: 8748–8763.
[31]	YANG Jianing, CHEN Xuweiyi, QIAN Shengyi, et al. LLM-grounder: Open-vocabulary 3D visual grounding with large language model as an agent[J]. arXiv preprint arXiv: 2309.12311, 2023. doi: 10.48550/arXiv.2309.12311.