Neural Surface Reconstruction and Rendering for LiDAR-Visual Systems

Jianheng Liu, Chunran Zheng, Yunfei Wan, Bowen Wang, Yixi Cai and Fu Zhang,
The University of Hong Kong,
Paper arXiv Video Code

Abstract

A MultiModal Mapping (M2Mapping) Framework for LiDAR-Visual Systems

This paper presents a unified surface reconstruction and rendering framework for LiDAR-visual systems, integrating Neural Radiance Fields (NeRF) and Neural Distance Fields (NDF) to recover both appearance and structural information from posed images and point clouds. We address the structural visible gap between NeRF and NDF by utilizing a visible-aware occupancy map to classify space into the free, occupied, visible unknown, and background regions. This classification facilitates the recovery of a complete appearance and structure of the scene. We unify the training of the NDF and NeRF using a spatial-varying scale SDF-to-density transformation for levels of detail for both structure and appearance. The proposed method leverages the learned NDF for structure-aware NeRF training by an adaptive sphere tracing sampling strategy for accurate structure rendering. In return, NeRF further refines structural in recovering missing or fuzzy structures in the NDF. Extensive experiments demonstrate the superior quality and versatility of the proposed method across various scenarios.

Video

Extrapolation Views Rendering Comparisions

3D Gaussian Splatting initialized with dense point clouds, denoted as 3DGS*

Replica

We emphasize the issues of extrapolation rendering consistency by uniformly sampling positions and orientations in each scene to generate the extrapolation dataset from Replica.

Ours
Ground Truth
H2Mapping
MonoGS
3DGS*
InstantNGP

FAST-LIVO2

Ours

3DGS*

InstantNGP

Surface Reconstruction Results

Compressed Mesh.

Replica

We show our surface reconstructions results on room-scale indoor simulation dataset.

Office-0

Office-1

Office-2

Office-3

Office-4

Room-0

Room-1

Room-2

FAST-LIVO2

For real-world datasets, we evaluate three types of trajectory datasets collected with a camera and a LiDAR: forward-facing, object-centric, and free-view trajectories. Using handheld devices and the FAST-LIVO2 localization algorithm, M2Mapping can be quickly deployed without time-consuming Structure-from-Motion.

Campus

Sculpture

Culture

Drive

BibTeX


          @misc{liu2024neuralsurfacereconstructionrendering,
            title={Neural Surface Reconstruction and Rendering for LiDAR-Visual Systems}, 
            author={Jianheng Liu and Chunran Zheng and Yunfei Wan and Bowen Wang and Yixi Cai and Fu Zhang},
            year={2024},
            eprint={2409.05310},
            archivePrefix={arXiv},
            primaryClass={cs.RO},
            url={https://arxiv.org/abs/2409.05310}, 
          }