Large scaling factor depth map super-resolution using progressive joint-multilateral filtering

Abstract

Depth images captured by conventional RGB-D sensors such as ToF cameras have limited resolution. Despite recent advances in depth camera technology, there is still a significant difference between the resolution of depth and color images. Therefore, depth map Super-Resolution (SR) techniques have received attention. Specifically, achieving an algorithm performing well at large scaling factors is of great importance and also challenging. In most existing methods, the up-sampling of low resolution depth images to the desired size is performed by an interpolation operation during the beginning stage and quality improvement filters are applied then. Due to the different nature of depth images and their sparsity, magnifying the images in a single step brings heavy artifacts specially at large up-sampling factors (e.g., 16). To tackle this problem, we propose a progressive multi-step depth map SR method where interpolation and modified enhancement processes are applied iteratively. This extremely improves the quality of the output depth image. Moreover, considering the importance of edges and discontinuities in depth images, instead of using conventional symmetric kernel, an edge directed kernel is applied which effectively avoids blurring. In addition, texture copying and depth bleeding artifacts are reduced employing a depth range filter. Quantitative and qualitative results of comprehensive experiments on Middlebury and real-world datasets demonstrate the effectiveness of our approach over prior depth SR works, especially for large scaling factors of 16, 32 and even 64. © 2022, The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature.