Abstract This article describes a technique to augment a typical RGBD sensor by integrating depth estimates obtained via Structure-from-Motion (SfM) with depth measurements from an RGBD sensor. Limitations in the RGBD depth sensing technology prevent capturing depth measurements in four important contexts: (1) distant surfaces (>8m), (2) dark surfaces, (3) brightly lit indoor scenes and (4) sunlit outdoor scenes. SfM technology computes depth via multi-view reconstruction from the RGB image sequence alone. As such, SfM depth estimates do not suffer the same limitations and may be computed in all four of the previously listed circumstances. This work describes a novel fusion of RGBD depth data and SfM-estimated depths to generate an improved depth stream that may be processed by one of many important downstream applications such as robot localization, robot mapping, robot navigation, object tracking, pose estimation, and object recognition.This approach is demonstrated on sequences of images that transition from indoor scenes, where the RGBD depth sensor can function, to outdoor scenes, where the RGBD depth sensor fails.
