“3D object manipulation in a single photograph using stock 3D models” by Kholgade, Simon, Efros and Sheikh

  • ©Natasha Kholgade, Tomas Simon, Alexei A. Efros, and Yaser Sheikh




    3D object manipulation in a single photograph using stock 3D models


Session Title: Image Tricks



    Photo-editing software restricts the control of objects in a photograph to the 2D image plane. We present a method that enables users to perform the full range of 3D manipulations, including scaling, rotation, translation, and nonrigid deformations, to an object in a photograph. As 3D manipulations often reveal parts of the object that are hidden in the original photograph, our approach uses publicly available 3D models to guide the completion of the geometry and appearance of the revealed areas of the object. The completion process leverages the structure and symmetry in the stock 3D model to factor out the effects of illumination, and to complete the appearance of the object. We demonstrate our system by producing object manipulations that would be impossible in traditional 2D photo-editing programs, such as turning a car over, making a paper-crane flap its wings, or manipulating airplanes in a historical photograph to change its story.


    1. Aubry, M., Maturana, D., Efros, A., Russell, B., and Sivic, J. 2014. Seeing 3D chairs: exemplar part-based 2D-3D alignment using a large dataset of CAD models. In Proc. CVPR.Google Scholar
    2. Avidan, S., and Shamir, A. 2007. Seam carving for content-aware image resizing. In Proc. ACM SIGGRAPH. Google ScholarDigital Library
    3. Barnes, C., Shechtman, E., Finkelstein, A., and Goldman, D. B. 2009. Patchmatch: a randomized correspondence algorithm for structural image editing. In Proc. ACM SIGGRAPH, 24:1–24:11. Google ScholarDigital Library
    4. Barrett, W. A., and Cheney, A. S. 2002. Object-based image editing. In Proc. ACM SIGGRAPH, 777–784. Google ScholarDigital Library
    5. Barron, J. T. 2012. Shape, albedo, and illumination from a single image of an unknown object. In CVPR, 334–341. Google ScholarDigital Library
    6. Blanz, V., and Vetter, T. 1999. A morphable model for the synthesis of 3d faces. In Proc. ACM SIGGRAPH, 187–194. Google ScholarDigital Library
    7. Bokeloh, M., Wand, M., Koltun, V., and Seidel, H.-P. 2011. Pattern-aware shape deformation using sliding dockers. ACM Trans. Graph. 30, 6 (Dec.), 123:1–123:10. Google ScholarDigital Library
    8. Chen, J., Paris, S., Wang, J., Matusik, W., Cohen, M., and Durand, F. 2011. The video mesh: A data structure for image-based three-dimensional video editing. In ICCP, 1–8.Google Scholar
    9. Chen, T., Zhu, Z., Shamir, A., Hu, S.-M., and Cohen-Or, D. 2013. 3-sweep: Extracting editable objects from a single photo. ACM Trans. Graph. 32, 6, to appear. Google ScholarDigital Library
    10. Debevec, P. E., Taylor, C. J., and Malik, J. 1996. Modeling and rendering architecture from photographs: a hybrid geometry- and image-based approach. In Proc. ACM SIGGRAPH, 11–20. Google ScholarDigital Library
    11. Debevec, P. 1998. Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography. In Proc. ACM SIGGRAPH, 189–198. Google ScholarDigital Library
    12. Durand, F. 2002. An invitation to discuss computer depiction. In Proc. ACM NPAR, 111–124. Google ScholarDigital Library
    13. Fang, H., and Hart, J. C. 2004. Textureshop: texture synthesis as a photograph editing tool. Proc. ACM SIGGRAPH, 354–359. Google ScholarDigital Library
    14. Fisher, R. 1953. Dispersion on a sphere. In Proceedings of the Royal Society of London. Series A, Mathematical and Physical Sciences, vol. 217, 295–305.Google ScholarCross Ref
    15. Gal, R., and Cohen-Or, D. 2006. Salient geometric features for partial shape matching and similarity. ACM Trans. Graph. 25, 1 (Jan.), 130–150. Google ScholarDigital Library
    16. Gal, R., Wexler, Y., Ofek, E., Hoppe, H., and Cohen-Or, D. 2010. Seamless montage for texturing models. Comput. Graph. Forum 29, 2, 479–486.Google ScholarCross Ref
    17. Goldberg, C., Chen, T., Zhang, F.-L., Shamir, A., and Hu, S.-M. 2012. Data-driven object manipulation in images. Computer Graphics Forum 31, 2pt1, 265–274. Google ScholarDigital Library
    18. Haber, T., Fuchs, C., Bekaert, P., Seidel, H.-P., Goesele, M., and Lensch, H. P. A. 2009. Relighting objects from image collections. In CVPR, IEEE, 627–634.Google Scholar
    19. Hara, K., Nishino, K., and Ikeuchi, K. 2008. Mixture of spherical distributions for single-view relighting. IEEE Trans. Pattern Anal. Mach. Intell. 30, 1, 25–35. Google ScholarDigital Library
    20. Hong, W., Yang, A. Y., Huang, K., and Ma, Y. 2004. On symmetry and multiple-view geometry: Structure, pose, and calibration from a single image. IJCV 60, 3 (Dec.), 241–265. Google ScholarDigital Library
    21. Karsch, K., Hedau, V., Forsyth, D., and Hoiem, D. 2011. Rendering synthetic objects into legacy photographs. In Proc. ACM SIGGRAPH Asia, 157:1–157:12. Google ScholarDigital Library
    22. Khan, E. A., Reinhard, E., Fleming, R. W., and Bülthoff, H. H. 2006. Image-based material editing. In Proc. ACM SIGGRAPH, 654–663. Google ScholarDigital Library
    23. Kim, V. G., Lipman, Y., and Funkhouser, T. 2012. Symmetry-guided texture synthesis and manipulation. ACM Trans. Graph. 31, 3 (June), 22:1–22:14. Google ScholarDigital Library
    24. Kolmogorov, V. 2006. Convergent tree-reweighted message passing for energy minimization. IEEE TPAMI 28, 10 (Oct.), 1568–1583. Google ScholarDigital Library
    25. Kopf, J., Neubert, B., Chen, B., Cohen, M., Cohen-Or, D., Deussen, O., Uyttendaele, M., and Lischinski, D. 2008. Deep photo: model-based photograph enhancement and viewing. In Proc. ACM SIGGRAPH Asia, 116:1–116:10. Google ScholarDigital Library
    26. Kraevoy, V., Sheffer, A., and Gotsman, C. 2003. Match-maker: constructing constrained texture maps. ACM Trans. Graph. 22, 3 (July), 326–333. Google ScholarDigital Library
    27. Kraevoy, V., Sheffer, A., and van de Panne, M. 2009. Modeling from contour drawings. In Proceedings of the 6th Eurographics Symposium on Sketch-Based Interfaces and Modeling, ACM, New York, NY, USA, SBIM ’09, 37–44. Google ScholarDigital Library
    28. Kwatra, V., Schodl, A., Essa, I., Turk, G., and Bobick, A. 2003. Graphcut textures: Image and video synthesis using graph cuts. ACM Trans. Graph. 22, 3 (July), 277–286. Google ScholarDigital Library
    29. Lalonde, J.-F., Hoiem, D., Efros, A. A., Rother, C., Winn, J., and Criminisi, A. 2007. Photo clip art. Proc. ACM SIGGRAPH 26, 3 (August), 3. Google ScholarDigital Library
    30. Land, E. H., John, and Mccann, J. 1971. Lightness and retinex theory. Journal of the Optical Society of America, 1–11.Google ScholarCross Ref
    31. Lepetit, V., Moreno-Noguer, F., and Fua, P. 2009. Epnp: An accurate o(n) solution to the pnp problem. Int. J. Comput. Vision 81, 155–166. Google ScholarDigital Library
    32. Lim, J., Piriavash, H., and Torralba, A. 2013. Parsing ikea objects: Fine pose estimation. In ICCV. Google ScholarDigital Library
    33. Mei, X., Ling, H., and Jacobs, D. 2009. Sparse representation of cast shadows via l1-regularized least squares. In ICCV.Google Scholar
    34. Mitra, N. J., and Pauly, M. 2008. Symmetry for architectural design. In Advances in Architectural Geometry, 13–16.Google Scholar
    35. Mitra, N. J., Guibas, L. J., and Pauly, M. 2006. Partial and approximate symmetry detection for 3d geometry. In Proc. ACM SIGGRAPH, 560–568. Google ScholarDigital Library
    36. Nealen, A., Sorkine, O., Alexa, M., and Cohen-Or, D. 2005. A sketch-based interface for detail-preserving mesh editing. ACM Trans. Graph. 24, 3 (July), 1142–1147. Google ScholarDigital Library
    37. Ng, R., Ramamoorthi, R., and Hanrahan, P. 2003. All-frequency shadows using non-linear wavelet lighting approximation. In Proc. ACM SIGGRAPH, 376–381. Google ScholarDigital Library
    38. Oh, B. M., Chen, M., Dorsey, J., and Durand, F. 2001. Image-based modeling and photo editing. In Proc. ACM SIGGRAPH, 433–442. Google ScholarDigital Library
    39. Okabe, T., Sato, I., and Sato, Y. 2004. Spherical harmonics vs. haar wavelets: Basis for recovering illumination from cast shadows. In CVPR, 50–57.Google Scholar
    40. Panagopoulos, A., Samaras, D., and Paragios, N. 2009. Robust shadow and illumination estimation using a mixture model. In CVPR, 651–658.Google Scholar
    41. Pauly, M., Mitra, N. J., Giesen, J., Gross, M., and Guibas, L. J. 2005. Example-based 3d scan completion. In Proc. SGP. Google ScholarDigital Library
    42. Prasad, M., Zisserman, A., and Fitzgibbon, A. W. 2006. Single view reconstruction of curved surfaces. In CVPR. Google ScholarDigital Library
    43. Ramamoorthi, R., and Hanrahan, P. 2001. On the relationship between radiance and irradiance: determining the illumination from images of a convex lambertian object. J. Opt. Soc. Am. A 18, 10, 2448–2459.Google ScholarCross Ref
    44. Romeiro, F., and Zickler, T. 2010. Blind reflectometry. In ECCV, 45–58. Google ScholarDigital Library
    45. Simpson, J., 2003. Oxford English Dictionary Online, 2nd edition. http://www.oed.com/, July.Google Scholar
    46. Sorkine, O., and Alexa, M. 2007. As-rigid-as-possible surface modeling. In Proc. SGP, 109–116. Google ScholarDigital Library
    47. Terzopoulos, D., Witkin, A., and Kass, M. 1987. Symmetry-seeking models and 3d object reconstruction. International Journal of Computer Vision 1, 211–221.Google ScholarCross Ref
    48. Tzur, Y., and Tal, A. 2009. Flexistickers: photogrammetric texture mapping using casual images. ACM Trans. Graph. 28, 3 (July), 45:1–45:10. Google ScholarDigital Library
    49. Xu, K., Zheng, H., Zhang, H., Cohen-Or, D., Liu, L., and Xiong, Y. 2011. Photo-inspired model-driven 3d object modeling. ACM Transactions on Graphics 30, 4. Google ScholarDigital Library
    50. Zheng, Y., Chen, X., Cheng, M.-M., Zhou, K., Hu, S.-M., and Mitra, N. J. 2012. Interactive images: cuboid proxies for smart image manipulation. ACM Trans. Graph. 31, 4 (July), 99:1–99:11. Google ScholarDigital Library
    51. Zou, H., and Hastie, T. 2005. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society, Series B 67, 301–320.Google ScholarCross Ref

ACM Digital Library Publication: