“Shading-based refinement on volumetric signed distance functions”

  • ©Michael Zollhöfer, Angela Dai, Matthias Innmann, Chenglei Wu, Marc Stamminger, Christian Theobalt, and Matthias Nießner

Conference:


Type:


Title:

    Shading-based refinement on volumetric signed distance functions

Session/Category Title: Geometry Zoo

Presenter(s)/Author(s):


Moderator(s):


Abstract:


    We present a novel method to obtain fine-scale detail in 3D reconstructions generated with low-budget RGB-D cameras or other commodity scanning devices. As the depth data of these sensors is noisy, truncated signed distance fields are typically used to regularize out the noise, which unfortunately leads to over-smoothed results. In our approach, we leverage RGB data to refine these reconstructions through shading cues, as color input is typically of much higher resolution than the depth data. As a result, we obtain reconstructions with high geometric detail, far beyond the depth resolution of the camera itself. Our core contribution is shading-based refinement directly on the implicit surface representation, which is generated from globally-aligned RGB-D images. We formulate the inverse shading problem on the volumetric distance field, and present a novel objective function which jointly optimizes for fine-scale surface geometry and spatially-varying surface reflectance. In order to enable the efficient reconstruction of sub-millimeter detail, we store and process our surface using a sparse voxel hashing scheme which we augment by introducing a grid hierarchy. A tailored GPU-based Gauss-Newton solver enables us to refine large shape models to previously unseen resolution within only a few seconds.

References:


    1. Agarwal, S., Furukawa, Y., Snavely, N., Simon, I., Curless, B., Seitz, S. M., and Szeliski, R. 2011. Building rome in a day. Communications of the ACM 54, 10, 105–112. Google ScholarDigital Library
    2. AgiSoft, L. 2014. Agisoft photoscan. Professional Edition.Google Scholar
    3. Beeler, T., Bradley, D., Zimmer, H., and Gross, M. 2012. Improved reconstruction of deforming surfaces by cancelling ambient occlusion. In Proc. ECCV, 30–43. Google ScholarDigital Library
    4. Bylow, E., Sturm, J., Kerl, C., Kahl, F., and Cremers, D. 2013. Real-time camera tracking and 3d reconstruction using signed distance functions. In Robotics: Science and Systems (RSS) Conference 2013, vol. 9.Google Scholar
    5. Carr, J. C., Beatson, R. K., Cherrie, J. B., Mitchell, T. J., Fright, W. R., McCallum, B. C., and Evans, T. R. 2001. Reconstruction and representation of 3d objects with radial basis functions. In Proc. SIGGRAPH, ACM, 67–76. Google ScholarDigital Library
    6. Chan, D., Buisman, H., Theobalt, C., Thrun, S., et al. 2008. A noise-aware filter for real-time depth upsampling. In Workshop on Multi-camera and Multi-modal Sensor Fusion Algorithms and Applications-M2SFA2 2008.Google Scholar
    7. Chen, Q., and Koltun, V. 2013. A simple model for intrinsic image decomposition with depth cues. In The IEEE International Conference on Computer Vision (ICCV). Google ScholarDigital Library
    8. Chen, J., Bautembach, D., and Izadi, S. 2013. Scalable real-time volumetric surface reconstruction. ACM TOG 32, 4, 113. Google ScholarDigital Library
    9. Cui, Y., Schuon, S., Chan, D., Thrun, S., and Theobalt, C. 2010. 3d shape scanning with a time-of-flight camera. In Proc. CVPR, 1173–1180.Google Scholar
    10. Curless, B., and Levoy, M. 1996. A volumetric method for building complex models from range images. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, ACM, 303–312. Google ScholarDigital Library
    11. Debevec, P. 2012. The light stages and their applications to photoreal digital actors. In SIGGRAPH Asia Technical Briefs.Google Scholar
    12. Diebel, J., and Thrun, S. 2006. An application of Markov Random Fields to range sensing. 291–298.Google Scholar
    13. Dolson, J., Baek, J., Plagemann, C., and Thrun, S. 2010. Upsampling range data in dynamic environments. In Proc. CVPR, IEEE, 1141–1148.Google Scholar
    14. Fuhrmann, S., and Goesele, M. 2014. Floating scale surface reconstruction. ACM TOG 33, 4, 46. Google ScholarDigital Library
    15. Ghosh, A., Fyffe, G., Tunwattanapong, B., Busch, J., Yu, X., and Debevec, P. 2011. Multiview face capture using polarized spherical gradient illumination. ACM TOG 30. Google ScholarDigital Library
    16. Goldluecke, B., Aubry, M., Kolev, K., and Cremers, D. 2014. A super-resolution framework for high-accuracy multiview reconstruction. ijcv 106, 2 (jan), 172–191. Google ScholarDigital Library
    17. Haber, T., Fuchs, C., Bekaer, P., Seidel, H.-P., Goesele, M., and Lensch, H. 2009. Relighting objects from image collections. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, 627–634.Google Scholar
    18. Han, Y., Lee, J.-Y., and Kweon, I. S. 2013. High quality shape from a single rgb-d image under uncalibrated natural illumination. In Proc. ICCV. Google ScholarDigital Library
    19. Hasinoff, S., Levin, A., Goode, P., and Freeman, W. 2011. Diffuse reflectance imaging with astronomical applications. In Proc. ICCV, 185–192. Google ScholarDigital Library
    20. Henry, P., Krainin, M., Herbst, E., Ren, X., and Fox, D. 2012. RGB-D mapping: Using Kinect-style depth cameras for dense 3D modeling of indoor environments. Int. J. Robotics Research 31 (apr), 647–663. Google ScholarDigital Library
    21. Hernández, C., Vogiatzis, G., and Cipolla, R. 2008. Multiview photometric stereo. IEEE PAMI 30, 3, 548–554. Google ScholarDigital Library
    22. Hoppe, H., DeRose, T., Duchamp, T., McDonald, J., and Stuetzle, W. 1992. Surface reconstruction from unorganized points. In Proceedings of the 19th Annual Conference on Computer Graphics and Interactive Techniques, ACM, New York, NY, USA, SIGGRAPH ’92, 71–78. Google ScholarDigital Library
    23. Horn, B. K. 1975. Obtaining shape from shading information. The psychology of computer vision, 115–155.Google Scholar
    24. Izadi, S., Kim, D., Hilliges, O., Molyneaux, D., Newcombe, R., Kohli, P., Shotton, J., Hodges, S., Freeman, D., Davison, A., et al. 2011. Kinectfusion: real-time 3d reconstruction and interaction using a moving depth camera. In Proc. UIST, ACM, 559–568. Google ScholarDigital Library
    25. Kazhdan, M., Bolitho, M., and Hoppe, H. 2006. Poisson surface reconstruction. In Proc. SGP. Google ScholarDigital Library
    26. Kehl, W., Navab, N., and Ilic, S. 2014. Coloured signed distance fields for full 3d object reconstruction. In Proc. BMVC.Google Scholar
    27. Keller, M., Lefloch, D., Lambers, M., Izadi, S., Weyrich, T., and Kolb, A. 2013. Real-time 3d reconstruction in dynamic scenes using point-based fusion. In Proc. 3DV, 1–8. Google ScholarDigital Library
    28. Kopf, J., Cohen, M. F., Lischinski, D., and Uyttendaele, M. 2007. Joint bilateral upsampling. ACM TOG 26, 3. Google ScholarDigital Library
    29. Lee, K. J., Zhao, Q., Tong, X., Gong, M., Izadi, S., Lee, S. U., Tan, P., and Lin, S. 2012. Estimation of intrinsic image sequences from image+depth video. In Proc. ECCV, 327–340. Google ScholarDigital Library
    30. Levoy, M., Pulli, K., Curless, B., Rusinkiewicz, S., Koller, D., Pereira, L., Ginzton, M., Anderson, S., Davis, J., Ginsberg, J., Shade, J., and Fulk, D. 2000. The digital michelangelo project: 3d scanning of large statues. In Proc. SIGGRAPH, 131–144. Google ScholarDigital Library
    31. Lindner, M., Kolb, A., and Hartmann, K. 2007. Data-fusion of PMD-based distance-information and high-resolution RGB-images. In Proc. ISSCS, 121–124.Google Scholar
    32. Lowe, D. G. 2004. Distinctive image features from scale-invariant keypoints. IJCV 60, 2, 91–110. Google ScholarDigital Library
    33. Mulligan, J., and Brolly, X. 2004. Surface determination by photometric ranging. In Proc. CVPR Workshops. Google ScholarDigital Library
    34. Nair, R., Ruhl, K., Lenzen, F., Meister, S., Schäfer, H., Garbe, C. S., Eisemann, M., Magnor, M., and Kondermann, D. 2013. A survey on time-of-flight stereo fusion. In Time-of-Flight and Depth Imaging. Sensors, Algorithms, and Applications. Springer, 105–127.Google Scholar
    35. Nehab, D., Rusinkiewicz, S., Davis, J., and Ramamoorthi, R. 2005. Efficiently combining positions and normals for precise 3D geometry. Proc. SIGGRAPH 24, 3. Google ScholarDigital Library
    36. Newcombe, R. A., and Davison, A. J. 2010. Live dense reconstruction with a single moving camera. In Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on, IEEE, 1498–1505.Google Scholar
    37. Newcombe, R. A., Davison, A. J., Izadi, S., Kohli, P., Hilliges, O., Shotton, J., Molyneaux, D., Hodges, S., Kim, D., and Fitzgibbon, A. 2011. Kinectfusion: Real-time dense surface mapping and tracking. In Proc. ISMAR, IEEE, 127–136. Google ScholarDigital Library
    38. Niessner, M., Zollhöfer, M., Izadi, S., and Stamminger, M. 2013. Real-time 3d reconstruction at scale using voxel hashing. ACM TOG 32, 6, 169. Google ScholarDigital Library
    39. Park, J., Kim, H., Tai, Y.-W., Brown, M. S., and Kweon, I.-S. 2011. High quality depth map upsampling for 3d-tof cameras. In Proc. ICCV, IEEE, 1623–1630. Google ScholarDigital Library
    40. Pradeep, V., Rhemann, C., Izadi, S., Zach, C., Bleyer, M., and Bathiche, S. 2013. Monofusion: Real-time 3d reconstruction of small scenes with a single web camera. In Proc. ISMAR.Google Scholar
    41. Prados, E., and Faugeras, O. 2005. Shape from shading: a well-posed problem? In Proc. CVPR. Google ScholarDigital Library
    42. Ramamoorthi, R., and Hanrahan, P. 2001. A signal-processing framework for inverse rendering. In Proc. SIGGRAPH, ACM, 117–128. Google ScholarDigital Library
    43. Richardt, C., Stoll, C., Dodgson, N. A., Seidel, H.-P., and Theobalt, C. 2012. Coherent spatiotemporal filtering, upsampling and rendering of RGBZ videos. CGF (Proceedings of Eurographics) 31, 2 (May). Google ScholarDigital Library
    44. Roth, H., and Vona, M. 2012. Moving volume kinectfusion. In BMVC, 1–11.Google Scholar
    45. Rusinkiewicz, S., Hall-Holt, O., and Levoy, M. 2002. Real-time 3D model acquisition. ACM TOG 21, 3, 438–446. Google ScholarDigital Library
    46. Scharstein, D., Hirschmüller, H., Kitajima, Y., Krathwohl, G., Nešić, N., Wang, X., and Westling, P. 2014. High-resolution stereo datasets with subpixel-accurate ground truth. In Pattern Recognition. Springer, 31–42.Google Scholar
    47. Schuon, S., Theobalt, C., Davis, J., and Thrun, S. 2009. Lidarboost: Depth superresolution for tof 3d shape scanning. In Proc. CVPR, IEEE, 343–350.Google Scholar
    48. Seitz, S., Curless, B., Diebel, J., Scharstein, D., and Szeliski, R. 2006. A comparison and evaluation of multi-view stereo reconstruction algorithms. In Proc. CVPR, vol. 1, 519–528. Google ScholarDigital Library
    49. Snavely, N., Seitz, S. M., and Szeliski, R. 2006. Photo tourism: exploring photo collections in 3d. ACM TOG 25, 3, 835–846. Google ScholarDigital Library
    50. Triggs, B., Mclauchlan, P. F., Hartley, R. I., and Fitzgibbon, A. W. 2000. Bundle adjustment–a modern synthesis. In Vision algorithms: theory and practice. Springer, 298–372. Google ScholarDigital Library
    51. Weber, D., Bender, J., Schnoes, M., Stork, A., and Fellner, D. 2013. Efficient gpu data structures and methods to solve sparse linear systems in dynamics applications. In CGF, vol. 32, Wiley Online Library, 16–26.Google Scholar
    52. Weise, T., Wismer, T., Leibe, B., and Van Gool, L. 2009. In-hand scanning with online loop closure. In ICCV Workshops, 1630–1637.Google ScholarCross Ref
    53. Whelan, T., Kaess, M., Fallon, M., Johannsson, H., Leonard, J., and McDonald, J. 2012. Kintinuous: Spatially extended kinectfusion.Google Scholar
    54. Wu, C., Varanasi, K., Liu, Y., Seidel, H.-P., and Theobalt, C. 2011. Shading-based dynamic shape refinement from multi-view video under general illumination. In Proc. ICCV, IEEE, 1108–1115. Google ScholarDigital Library
    55. Wu, C., Stoll, C., Valgaerts, L., and Theobalt, C. 2013. On-set performance capture of multiple actors with a stereo camera. ACM TOG (Proc. SIGGRAPh Asia) 32, 6, 161. Google ScholarDigital Library
    56. Wu, C., Zollhöfer, M., Niessner, M., Stamminger, M., Izadi, S., and Theobalt, C. 2014. Real-time shading-based refinement for consumer depth cameras. ACM TOG (Proc. SIGGRAPH Asia) 33, 6, 200:1–200:10. Google ScholarDigital Library
    57. Yu, L.-F., Yeung, S.-K., Tai, Y.-W., and Lin, S. 2013. Shading-based shape refinement of rgb-d images. In Proc. CVPR. Google ScholarDigital Library
    58. Zhang, Z., Tsa, P.-S., Cryer, J. E., and Shah, M. 1999. Shape from shading: A survey. IEEE PAMI 21, 8, 690–706. Google ScholarDigital Library
    59. Zhou, Q.-Y., and Koltun, V. 2014. Color map optimization for 3d reconstruction with consumer depth cameras. ACM Transactions on Graphics (TOG) 33, 4, 155. Google ScholarDigital Library
    60. Zollhöfer, M., Niessner, M., Izadi, S., Rehmann, C., Zach, C., Fisher, M., Wu, C., Fitzgibbon, A., Loop, C., Theobalt, C., et al. 2014. Real-time non-rigid reconstruction using an rgb-d camera. ACM TOG (Proc. SIGGRAPH) 4. Google ScholarDigital Library

ACM Digital Library Publication:


Overview Page: