Non-Glare-NeRF

Abstract

This paper proposes two approaches to solve the depth distortion problem that occurs on glare surfaces in the process of 3D reconstruction based on Neural Radiance Fields (NeRF).

First, by introducing a preprocessing process using the Inpainting technique, distortion of texture and depth information due to Glare is alleviated. Second, a dataset that combines high-precision depth information and rich color information is constructed using the iPhone's LiDAR sensor and RGB camera.

It was confirmed through experiments that the proposed method improves the 3D reconstruction performance of NeRF even in scenes where the Glare phenomenon exists.

Video

Non-Glare-NeRF Methods

Pipeline of Non-Glare-NeRF: (a): Extract Instant NGP (NeRF) model results. (b): Extract the result of Instant NGP (NeRF) model with pre-processing. Pre-processing includes the process of detecting glare areas and applying interpainting.

How to Create Depth Dataset using iphone

Data Pipeline for Depth Evaluation: Using the iPhone's LiDAR sensor and RGB camera, a dataset including accurate depth information and color information was constructed. (a) Data Collection: I used my iPhone to shoot certain scenes, and I extracted RGB images and camera poses for each frame. (b) Acquiring depth information: As shown in Figure, depth information obtained from the LiDAR sensor was combined with RGB images to create the Depth Ground Truth. At this time, the Depth Completion technique was applied to supplement the gaps in the LiDAR data.

Dataset Construction: This diagram illustrates the process of creating a 3D dataset using ARFrame and ARDepthData. (i) ARFrame represents frame-based data provided by ARKit, collected in real-time from devices like iPhones. (ii) ARDepthData represents depth information collected by the LiDAR sensor in the iPhone. ARKit generates ARDepthData based on sceneDepth, which includes two key components: - depthMap: A map containing depth values (z-values) for each pixel. - confidenceMap: A map representing the confidence level of the depth data, typically categorized into low, middle, and high levels.
Each point in the depthMap is transformed into its corresponding 3D position in space. The confidenceMap can be used to filter points with low accuracy, ensuring better results.

Dataset Construction: The dataset was constructed by combining the RGB image and the depth information obtained from the iPhone's LiDAR sensor.

Rendering Results

Visualization: Here are examples of HuGS on different scenes (datasets). More results can be found in the paper and the data.

(1) Input RGB

(2) Input Depth

(3) Glare Mask

(4) Inpainting

(5) Result RGB

(6) Result Depth

Quality Evaluation: Our methodology shows that the Glare region is effectively restored, and we obtain much more natural texture and depth information compared to existing methods.

classroom1

Ground Truth

instant NGP

w/ Non-Glare-NeRF (ours)

classroom2

Qualitative Evaluation: The proposed method showed excellent overall performance and the performance improvement was particularly remarkable in the SqRel and RMSELog indicators. Through this, it was confirmed that the proposed method showed better performance in depth generation reflecting the scale.

BibTeX

@article{choi2024nonglarenerf,
  author    = {Choi, Seongjun and Hwang, Hyoseok},
  title     = {Non-Glare-NeRF: To prevent unwanted glare},
  journal   = {KSC},
  year      = {2024},
}