“Color map optimization for 3D reconstruction with consumer depth cameras” by Zhou and Koltun

  • ©Qian-Yi Zhou and Vladlen Koltun

Conference:


Type:


Title:

    Color map optimization for 3D reconstruction with consumer depth cameras

Session/Category Title: Depth for All Occasions

Presenter(s)/Author(s):


Moderator(s):


Abstract:


    We present a global optimization approach for mapping color images onto geometric reconstructions. Range and color videos produced by consumer-grade RGB-D cameras suffer from noise and optical distortions, which impede accurate mapping of the acquired color data to the reconstructed geometry. Our approach addresses these sources of error by optimizing camera poses in tandem with non-rigid correction functions for all images. All parameters are optimized jointly to maximize the photometric consistency of the reconstructed mapping. We show that this optimization can be performed efficiently by an alternating optimization algorithm that interleaves analytical updates of the color map with decoupled parameter updates for all images. Experimental results demonstrate that our approach substantially improves color mapping fidelity.

References:


    1. Aganj, E., Monasse, P., and Keriven, R. 2009. Multi-view texturing of imprecise mesh. In ACCV. Google ScholarDigital Library
    2. Baumberg, A. 2002. Blending images for texturing 3D models. In BMVC.Google Scholar
    3. Bernardini, F., Martin, I. M., and Rushmeier, H. E. 2001. High-quality texture reconstruction from multiple scans. IEEE Transactions on Visualization and Computer Graphics 7, 4. Google ScholarDigital Library
    4. Bylow, E., Sturm, J., Kerl, C., Kahl, F., and Cremers, D. 2013. Real-time camera tracking and 3D reconstruction using signed distance functions. In RSS.Google Scholar
    5. Callieri, M., Cignoni, P., Corsini, M., and Scopigno, R. 2008. Masked photo blending: Mapping dense photographic data set on high-resolution sampled 3D models. Computers & Graphics 32, 3. Google ScholarDigital Library
    6. Chen, Q., and Koltun, V. 2013. A simple model for intrinsic image decomposition with depth cues. In ICCV. Google ScholarDigital Library
    7. Chen, J., Bautembach, D., and Izadi, S. 2013. Scalable real-time volumetric surface reconstruction. ACM Transactions on Graphics 32, 4. Google ScholarDigital Library
    8. Chuang, M., Luo, L., Brown, B. J., Rusinkiewicz, S., and Kazhdan, M. M. 2009. Estimating the Laplace-Beltrami operator by restricting 3D functions. Computer Graphics Forum 28, 5. Google ScholarDigital Library
    9. Corsini, M., Dellepiane, M., Ponchio, F., and Scopigno, R. 2009. Image-to-geometry registration: a mutual information method exploiting illumination-related geometric properties. Computer Graphics Forum 28, 7.Google ScholarCross Ref
    10. Corsini, M., Dellepiane, M., Ganovelli, F., Gherardi, R., Fusiello, A., and Scopigno, R. 2013. Fully automatic registration of image sets on approximate geometry. International Journal of Computer Vision 102, 1–3. Google ScholarDigital Library
    11. Crete, F., Dolmiere, T., Ladret, P., and Nicolas, M. 2007. The blur effect: perception and estimation with a new no-reference perceptual blur metric. In SPIE.Google Scholar
    12. Dellepiane, M., Marroquim, R., Callieri, M., Cignoni, P., and Scopigno, R. 2012. Flow-based local optimization for image-to-geometry projection. IEEE Transactions on Visualization and Computer Graphics 18, 3. Google ScholarDigital Library
    13. Endres, F., Hess, J., Sturm, J., Cremers, D., and Burgard, W. 2014. 3D mapping with an RGB-D camera. IEEE Transactions on Robotics 30, 1. Google ScholarDigital Library
    14. Franken, T., Dellepiane, M., Ganovelli, F., Cignoni, P., Montani, C., and Scopigno, R. 2005. Minimizing user intervention in registering 2D images to 3D models. The Visual Computer 21, 8–10.Google ScholarCross Ref
    15. Gal, R., Wexler, Y., Ofek, E., Hoppe, H., and Cohen-Or, D. 2010. Seamless montage for texturing models. Computer Graphics Forum 29, 2.Google ScholarCross Ref
    16. Ikeuchi, K., Oishi, T., Takamatsu, J., Sagawa, R., Nakazawa, A., Kurazume, R., Nishino, K., Kamakura, M., and Okamoto, Y. 2007. The great Buddha project: Digitally archiving, restoring, and analyzing cultural heritage objects. International Journal of Computer Vision 75, 1. Google ScholarDigital Library
    17. Izadi, S., Kim, D., Hilliges, O., Molyneaux, D., Newcombe, R. A., Kohli, P., Shotton, J., Hodges, S., Freeman, D., Davison, A. J., and Fitzgibbon, A. W. 2011. KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera. In UIST. Google ScholarDigital Library
    18. Johnson, A. E., and Kang, S. B. 1999. Registration and integration of textured 3D data. Image and Vision Computing 17, 2.Google ScholarCross Ref
    19. Laffont, P.-Y., Bousseau, A., and Drettakis, G. 2013. Rich intrinsic image decomposition of outdoor scenes from multiple views. IEEE Transactions on Visualization and Computer Graphics 19, 2. Google ScholarDigital Library
    20. Lempitsky, V. S., and Ivanov, D. V. 2007. Seamless mosaicing of image-based texture maps. In CVPR.Google Scholar
    21. Lensch, H. P. A., Heidrich, W., and Seidel, H.-P. 2001. A silhouette-based algorithm for texture registration and stitching. Graphical Models 63, 4. Google ScholarDigital Library
    22. Levoy, M., Pulli, K., Curless, B., Rusinkiewicz, S., Koller, D., Pereira, L., Ginzton, M., Anderson, S., Davis, J., Ginsberg, J., Shade, J., and Fulk, D. 2000. The digital Michelangelo project: 3D scanning of large statues. In SIGGRAPH. Google ScholarDigital Library
    23. Li, H., Vouga, E., Gudym, A., Luo, L., Barron, J. T., and Gusev, G. 2013. 3D self-portraits. ACM Transactions on Graphics 32, 6. Google ScholarDigital Library
    24. Liu, L., and Stamos, I. 2012. A systematic approach for 2D-image to 3D-range registration in urban environments. Computer Vision and Image Understanding 116, 1. Google ScholarDigital Library
    25. Neugebauer, P. J., and Klein, K. 1999. Texturing 3D models of real world objects from multiple unregistered photographic views. Computer Graphics Forum 18, 3.Google ScholarCross Ref
    26. Newcombe, R. A., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A. J., Kohli, P., Shotton, J., Hodges, S., and Fitzgibbon, A. 2011. KinectFusion: Real-time dense surface mapping and tracking. In ISMAR. Google ScholarDigital Library
    27. Niessner, M., Zollhöfer, M., Izadi, S., and Stamminger, M. 2013. Real-time 3D reconstruction at scale using voxel hashing. ACM Transactions on Graphics 32, 6. Google ScholarDigital Library
    28. Ofek, E., Shilat, E., Rappoport, A., and Werman, M. 1997. Multiresolution textures from image sequences. IEEE Computer Graphics and Applications 17, 2. Google ScholarDigital Library
    29. Peters, J., and Reif, U. 1997. The simplest subdivision scheme for smoothing polyhedra. ACM Transactions on Graphics 16, 4. Google ScholarDigital Library
    30. Pighin, F. H., Hecker, J., Lischinski, D., Szeliski, R., and Salesin, D. 1998. Synthesizing realistic facial expressions from photographs. In SIGGRAPH. Google ScholarDigital Library
    31. Pulli, K., and Shapiro, L. G. 2000. Surface reconstruction and display from range and color data. Graphical Models 62, 3. Google ScholarDigital Library
    32. Pulli, K., Piiroinen, S., Duchamp, T., and Stuetzle, W. 2005. Projective surface matching of colored 3D scans. In 3DIM. Google ScholarDigital Library
    33. Rocchini, C., Cignoni, P., Montani, C., and Scopigno, R. 1999. Multiple texture stitching and blending on 3D objects. In Rendering Techniques. Google ScholarDigital Library
    34. Rusu, R. B., and Cousins, S. 2011. 3D is here: Point Cloud Library (PCL). In ICRA.Google Scholar
    35. Shan, Q., Adams, R., Curless, B., Furukawa, Y., and Seitz, S. M. 2013. The visual Turing test for scene reconstruction. In 3DV. Google ScholarDigital Library
    36. Sinha, S. N., Steedly, D., Szeliski, R., Agrawala, M., and Pollefeys, M. 2008. Interactive 3D architectural modeling from unordered photo collections. ACM Transactions on Graphics 27, 5. Google ScholarDigital Library
    37. Stamos, I., and Allen, P. K. 2000. 3-D model construction using range and image data. In CVPR.Google Scholar
    38. Stamos, I., and Allen, P. K. 2002. Geometry and texture recovery of scenes of large scale. Computer Vision and Image Understanding 88, 2. Google ScholarDigital Library
    39. Sturm, J., Bylow, E., Kahl, F., and Cremers, D. 2013. CopyMe3D: Scanning and printing persons in 3D. In GCPR.Google Scholar
    40. Troccoli, A., and Allen, P. K. 2008. Building illumination coherent 3D models of large-scale outdoor scenes. International Journal of Computer Vision 78, 2–3. Google ScholarDigital Library
    41. Weyrich, T., Lawrence, J., Lensch, H. P. A., Rusinkiewicz, S., and Zickler, T. 2009. Principles of appearance acquisition and representation. Foundations and Trends in Computer Graphics and Vision 4, 2. Google ScholarDigital Library
    42. Whelan, T., Johannsson, H., Kaess, M., Leonard, J., and McDonald, J. 2013. Robust real-time visual odometry for dense RGB-D mapping. In ICRA.Google Scholar
    43. Yamauchi, H., Lensch, H. P. A., Haber, J., and Seidel, H.-P. 2005. Textures revisited. The Visual Computer 21, 4. Google ScholarDigital Library
    44. Zhou, Q.-Y., and Koltun, V. 2013. Dense scene reconstruction with points of interest. ACM Transactions on Graphics 32, 4. Google ScholarDigital Library
    45. Zhou, Q.-Y., Miller, S., and Koltun, V. 2013. Elastic fragments for dense scene reconstruction. In ICCV. Google ScholarDigital Library

ACM Digital Library Publication:


Overview Page: