Dealing With The Two Difficult Scenes In Image Based 3d Reconstruction

"Although image-based 3D reconstruction by photogrammetry has been well studied and applied, applying this technique in the scenarios that violate the classical shading assumptions can lead to poor performance. In this work we consider dealing with two kinds of difficult scenes that are currently problematic for 3D reconstruction: night scenes, and the glass reflection scenes. We mainly focus on the pre-processing or post-processing of the data in structure-from-motion (SFM) pipeline, and hence to improve the performance of the existing reconstruction algorithm in specific scenes.At night, low-light conditions often cause the images to lack sharpness, and high-dynamic range issues lead to saturation. The SFM reconstruction pipeline that works well in daylight is likely to recover only a limited quantity of dense points of bright fragmented objects near artificial lighting. Here, we verify that with ordinary camera images the default SFM procedures cannot generate the complete structure and the precise texture of the urban night scenery model. We develop a novel solution based on registration and synthesis between the night-time reconstruction and that of the same region in daytime. We implement a registration pipeline for conformal matching of the day and night point clouds. For the coarse registration step, we use detected plane features to search and match 4-plane congruent sets, and then refine the transformation matrix by minimizing the plane-plane errors and the point-plane errors. For the fine registration step, we consider the positions of windows, a commonly-occurring object cue in urban building scenes. The windows are segmented from each image using Mask R-CNN and the pixel-level masks are projected into the 3D space. Registration between the extracted windows and the night scenery is computed using iterative closest point method. This leads to final registration error less than 0.2 degrees in rotation, and 0.2% in translation and scale. Finally, we synthesize the daytime textured model and the night point clouds to produce vivid visual effects of the urban night scenery.For the scenes containing a partial mirror of glass which both transmits and reflects incident light, the mirror image can cause severe noise to the dense reconstruction process, because the blurred mirror image is detrimental to feature matching and densification, and compositing between the reflection and the transmission is essentially ambiguous for the ordinary SFM principle. As many previous researches have been focused on the image reflection removal algorithms, we propose to take advantage of a pre-processing step of the image sequence. The workflow starts from the output of the state-of-art deep learning model based on perceptual losses, and we feed the initial separation into a Markov Random Field (MRF) model to refine the results by maximizing the posteriori probability combining priors. The MRF is defined on a pair of the reflection and the transmission fields, and we construct a pairwise neighborhood system including both the spatial neighbors and the transmission-reflection neighbors. We perform inference on the MRF by belief propagation. An additional second refinement stage is applied on multiple images by registration between frame-to-frame to achieve further reflection components erasing. The workflow separates and removes the undesired reflection noise on stereo image sequences.Finally the output through the SFM pipeline shows effective elimination or suppression of the noise comparing to the baseline, where more than 90% of noise point clouds caused by the reflection are cleared. In general, our work provides a robust and feasible methodology in reconstructing the scene with glass mirror reflection"--