“Estimating image depth using shape collections” by Su, Huang, Mitra, Li and Guibas

  • ©Hao Su, Qixing Huang, Niloy J. Mitra, Yangyan Li, and Leonidas (Leo) J. Guibas




    Estimating image depth using shape collections


Session Title: Shape Collection



    Images, while easy to acquire, view, publish, and share, they lack critical depth information. This poses a serious bottleneck for many image manipulation, editing, and retrieval tasks. In this paper we consider the problem of adding depth to an image of an object, effectively ‘lifting’ it back to 3D, by exploiting a collection of aligned 3D models of related objects. Our key insight is that, even when the imaged object is not contained in the shape collection, the network of shapes implicitly characterizes a shape-specific deformation subspace that regularizes the problem and enables robust diffusion of depth information from the shape collection to the input image. We evaluate our fully automatic approach on diverse and challenging input images, validate the results against Kinect depth readings, and demonstrate several imaging applications including depth-enhanced image editing and image relighting.


    1. Averkiou, M., Kim, V., Zheng, Y., and Mitra, N. J. 2014. Shapesynth: Parameterizing model collections for coupled shape exploration and synthesis. CGF.Google Scholar
    2. Chaudhuri, S., Kalogerakis, E., Guibas, L., and Koltun, V. 2011. Probabilistic reasoning for assembly-based 3d modeling. ACM ToG 30, 4 (Aug.), 35:1–35:10. Google ScholarDigital Library
    3. Chen, D.-Y., Tian, X.-P., Shen, Y.-T., and Ouhyoung, M. 2003. On visual similarity based 3d model retrieval. CGF 22, 3, 223–232.Google ScholarCross Ref
    4. Coifman, R. R., Lafon, S., Lee, A. B., Maggioni, M., Nadler, B., Warner, F., and Zucker, S. W. 2005. Geometric diffusions as a tool for harmonic analysis and structure definition of data: Diffusion maps. PNAS 102, 21, 7426–7431.Google ScholarCross Ref
    5. Cyr, C. M., and Kimia, B. B. 2004. A similarity-based aspect-graph approach to 3d object recognition. IJCV 57, 1 (Apr.), 5–22. Google ScholarDigital Library
    6. Dalal, N., and Triggs, B. 2005. Histograms of oriented gradients for human detection. In Proc. CVPR, 886–893. Google ScholarDigital Library
    7. Funkhouser, T., Kazhdan, M., Shilane, P., Min, P., Kiefer, W., Tal, A., Rusinkiewicz, S., and Dobkin, D. 2004. Modeling by example. ACM ToG 23, 3 (Aug.), 652–663. Google ScholarDigital Library
    8. Goldman, D. B., Curless, B., Hertzmann, A., and Seitz, S. M. 2005. Shape and spatially-varying brdfs from photometric stereo. In Proc. ICCV, 341–348. Google ScholarDigital Library
    9. Hoiem, D., Efros, A. A., and Hebert, M. 2005. Automatic photo pop-up. ACM ToG 24, 3 (July), 577–584. Google ScholarDigital Library
    10. Huang, Q., and Guibas, L. 2013. Consistent shape maps via semidefinite programming. CGF 32, 5, 177–186. Google ScholarDigital Library
    11. Huang, Q., Koltun, V., and Guibas, L. 2011. Joint shape segmentation using linear programming. ACM ToG 30, 6. Google ScholarDigital Library
    12. Huang, Q.-X., Su, H., and Guibas, L. 2013. Fine-grained semi-supervised labeling of large shape collections. ACM ToG 32, 6 (Nov.), 190:1–190:10. Google ScholarDigital Library
    13. Kalogerakis, E., Chaudhuri, S., Koller, D., and Koltun, V. 2012. A probabilistic model for component-based shape synthesis. ACM ToG 31, 4 (July), 55:1–55:11. Google ScholarDigital Library
    14. Kazhdan, M., Bolitho, M., and Hoppe, H. 2006. Poisson surface reconstruction. CGF, 61–70. Google ScholarDigital Library
    15. Kim, V. G., Li, W., Mitra, N. J., DiVerdi, S., and Funkhouser, T. 2012. Exploring collections of 3d models using fuzzy correspondences. ACM ToG 31, 4 (July), 54:1–54:11. Google ScholarDigital Library
    16. Kim, Y. M., Mitra, N. J., Yan, D.-M., and Guibas, L. 2012. Acquiring 3d indoor environments with variability and repetition. ACM ToG 31, 6 (Nov.), 138:1–138:11. Google ScholarDigital Library
    17. Kim, V. G., Li, W., Mitra, N. J., Chaudhuri, S., DiVerdi, S., and Funkhouser, T. 2013. Learning part-based templates from large collections of 3d shapes. ACM ToG 32, 4, 70:1–70:12. Google ScholarDigital Library
    18. Lensch, H. P. A., Kautz, J., Goesele, M., Heidrich, W., and Seidel, H.-P. 2003. Image-based reconstruction of spatial appearance and geometric detail. ACM ToG 22, 2. Google ScholarDigital Library
    19. Marcato Jr, R. W. 1998. Optimizing an inverse warper. PhD thesis, Massachusetts Institute of Technology.Google Scholar
    20. Mitra, N. J., Guibas, L. J., and Pauly, M. 2006. Partial and approximate symmetry detection for 3d geometry. ACM ToG, 560–568. Google ScholarDigital Library
    21. Mitra, N. J., Guibas, L., and Pauly, M. 2007. Symmetrization. ACM ToG 26, 3, #63, 1–8. Google ScholarDigital Library
    22. Munich, M. E., and Perona, P. 1999. Continuous dynamic time warping for translation-invariant curve alignment with applications to signature verification. In ICCV, vol. 1.Google Scholar
    23. Nan, L., Xie, K., and Sharf, A. 2012. A search-classify approach for cluttered indoor scene understanding. ACM ToG 31, 6 (Nov.), 137:1–137:10. Google ScholarDigital Library
    24. Oliva, A., and Torralba, A. 2001. Modeling the shape of the scene: A holistic representation of the spatial envelope. IJCV 42, 3 (May), 145–175. Google ScholarDigital Library
    25. Oliva, A., and Torralba, A. 2006. Building the gist of a scene: The role of global image features in recognition. Progress in Brain Research 155, 23–36.Google ScholarCross Ref
    26. Osada, R., Funkhouser, T., Chazelle, B., and Dobkin, D. 2002. Shape distributions. ACM ToG 21 (October), 807–832. Google ScholarDigital Library
    27. Ovsjanikov, M., Li, W., Guibas, L., and Mitra, N. J. 2011. Exploration of continuous variability in collections of 3d shapes. ACM ToG 30, 4, 33:1–33:10. Google ScholarDigital Library
    28. Rusinkiewicz, S., and Levoy, M. 2001. Efficient variants of the ICP algorithm. In 3DIM, 145–152.Google Scholar
    29. Saxena, A., Sun, M., and Ng, A. Y. 2009. Make3d: Learning 3d scene structure from a single still image. IEEE TPAMI 31, 5 (May), 824–840. Google ScholarDigital Library
    30. Shen, C.-H., Fu, H., Chen, K., and Hu, S.-M. 2012. Structure recovery by part assembly. ACM ToG 31, 6, 180:1–180:11. Google ScholarDigital Library
    31. Sorkine, O., Cohen-Or, D., Lipman, Y., Alexa, M., Rössl, C., and Seidel, H.-P. 2004. Laplacian surface editing. CGF, 175–184. Google ScholarDigital Library
    32. Sumner, R. W., Schmid, J., and Pauly, M. 2007. Embedded deformation for shape manipulation. ACM ToG 26, 3 (July). Google ScholarDigital Library
    33. Sun, M., Kumar, S. S., Bradski, G., and Savarese, S. 2011. Toward automatic 3d generic object modeling from one single image. In 3DIMPVT, IEEE. Google ScholarDigital Library
    34. Szeliski, R., Zabih, R., Scharstein, D., Veksler, O., Kolmogorov, V., Agarwala, A., Tappen, M., and Rother, C. 2008. A comparative study of energy minimization methods for markov random fields with smoothness-based priors. IEEE TPAMI 30, 6 (June), 1068–1080. Google ScholarDigital Library
    35. Wang, Y., Gong, M., Wang, T., Cohen-Or, D., Zhang, H., and Chen, B. 2013. Projective analysis for 3d shape segmentation. ACM ToG 32, 6 (Nov.), 192:1–192:12. Google ScholarDigital Library
    36. Wu, T.-P., Sun, J., Tang, C.-K., and Shum, H.-Y. 2008. Interactive normal reconstruction from a single image. ACM ToG, 119:1–119:9. Google ScholarDigital Library
    37. Xu, K., Zheng, H., Zhang, H., Cohen-Or, D., Liu, L., and Xiong, Y. 2011. Photo-inspired model-driven 3d object modeling. ACM ToG 30, 4 (July), 80:1–80:10. Google ScholarDigital Library
    38. Zheng, Y., Chen, X., Cheng, M.-M., Zhou, K., Hu, S.-M., and Mitra, N. J. 2012. Interactive images: Cuboid proxies for smart image manipulation. ACM ToG 31, 4, 99:1–99:11. Google ScholarDigital Library
    39. Zia, Z., Stark, M., Schiele, B., and Schindler, K. 2013. Detailed 3d representations for object recognition and modeling. IEEE TPAMI 35, 11, 2608–2623. Google ScholarDigital Library

ACM Digital Library Publication: