“3D object manipulation in a single photograph using stock 3D models” by Kholgade, Simon, Efros and Sheikh
Conference:
Type:
Title:
- 3D object manipulation in a single photograph using stock 3D models
Session/Category Title: Image Tricks
Presenter(s)/Author(s):
Moderator(s):
Abstract:
Photo-editing software restricts the control of objects in a photograph to the 2D image plane. We present a method that enables users to perform the full range of 3D manipulations, including scaling, rotation, translation, and nonrigid deformations, to an object in a photograph. As 3D manipulations often reveal parts of the object that are hidden in the original photograph, our approach uses publicly available 3D models to guide the completion of the geometry and appearance of the revealed areas of the object. The completion process leverages the structure and symmetry in the stock 3D model to factor out the effects of illumination, and to complete the appearance of the object. We demonstrate our system by producing object manipulations that would be impossible in traditional 2D photo-editing programs, such as turning a car over, making a paper-crane flap its wings, or manipulating airplanes in a historical photograph to change its story.
References:
1. Aubry, M., Maturana, D., Efros, A., Russell, B., and Sivic, J. 2014. Seeing 3D chairs: exemplar part-based 2D-3D alignment using a large dataset of CAD models. In Proc. CVPR.Google Scholar
2. Avidan, S., and Shamir, A. 2007. Seam carving for content-aware image resizing. In Proc. ACM SIGGRAPH. Google ScholarDigital Library
3. Barnes, C., Shechtman, E., Finkelstein, A., and Goldman, D. B. 2009. Patchmatch: a randomized correspondence algorithm for structural image editing. In Proc. ACM SIGGRAPH, 24:1–24:11. Google ScholarDigital Library
4. Barrett, W. A., and Cheney, A. S. 2002. Object-based image editing. In Proc. ACM SIGGRAPH, 777–784. Google ScholarDigital Library
5. Barron, J. T. 2012. Shape, albedo, and illumination from a single image of an unknown object. In CVPR, 334–341. Google ScholarDigital Library
6. Blanz, V., and Vetter, T. 1999. A morphable model for the synthesis of 3d faces. In Proc. ACM SIGGRAPH, 187–194. Google ScholarDigital Library
7. Bokeloh, M., Wand, M., Koltun, V., and Seidel, H.-P. 2011. Pattern-aware shape deformation using sliding dockers. ACM Trans. Graph. 30, 6 (Dec.), 123:1–123:10. Google ScholarDigital Library
8. Chen, J., Paris, S., Wang, J., Matusik, W., Cohen, M., and Durand, F. 2011. The video mesh: A data structure for image-based three-dimensional video editing. In ICCP, 1–8.Google Scholar
9. Chen, T., Zhu, Z., Shamir, A., Hu, S.-M., and Cohen-Or, D. 2013. 3-sweep: Extracting editable objects from a single photo. ACM Trans. Graph. 32, 6, to appear. Google ScholarDigital Library
10. Debevec, P. E., Taylor, C. J., and Malik, J. 1996. Modeling and rendering architecture from photographs: a hybrid geometry- and image-based approach. In Proc. ACM SIGGRAPH, 11–20. Google ScholarDigital Library
11. Debevec, P. 1998. Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography. In Proc. ACM SIGGRAPH, 189–198. Google ScholarDigital Library
12. Durand, F. 2002. An invitation to discuss computer depiction. In Proc. ACM NPAR, 111–124. Google ScholarDigital Library
13. Fang, H., and Hart, J. C. 2004. Textureshop: texture synthesis as a photograph editing tool. Proc. ACM SIGGRAPH, 354–359. Google ScholarDigital Library
14. Fisher, R. 1953. Dispersion on a sphere. In Proceedings of the Royal Society of London. Series A, Mathematical and Physical Sciences, vol. 217, 295–305.Google ScholarCross Ref
15. Gal, R., and Cohen-Or, D. 2006. Salient geometric features for partial shape matching and similarity. ACM Trans. Graph. 25, 1 (Jan.), 130–150. Google ScholarDigital Library
16. Gal, R., Wexler, Y., Ofek, E., Hoppe, H., and Cohen-Or, D. 2010. Seamless montage for texturing models. Comput. Graph. Forum 29, 2, 479–486.Google ScholarCross Ref
17. Goldberg, C., Chen, T., Zhang, F.-L., Shamir, A., and Hu, S.-M. 2012. Data-driven object manipulation in images. Computer Graphics Forum 31, 2pt1, 265–274. Google ScholarDigital Library
18. Haber, T., Fuchs, C., Bekaert, P., Seidel, H.-P., Goesele, M., and Lensch, H. P. A. 2009. Relighting objects from image collections. In CVPR, IEEE, 627–634.Google Scholar
19. Hara, K., Nishino, K., and Ikeuchi, K. 2008. Mixture of spherical distributions for single-view relighting. IEEE Trans. Pattern Anal. Mach. Intell. 30, 1, 25–35. Google ScholarDigital Library
20. Hong, W., Yang, A. Y., Huang, K., and Ma, Y. 2004. On symmetry and multiple-view geometry: Structure, pose, and calibration from a single image. IJCV 60, 3 (Dec.), 241–265. Google ScholarDigital Library
21. Karsch, K., Hedau, V., Forsyth, D., and Hoiem, D. 2011. Rendering synthetic objects into legacy photographs. In Proc. ACM SIGGRAPH Asia, 157:1–157:12. Google ScholarDigital Library
22. Khan, E. A., Reinhard, E., Fleming, R. W., and Bülthoff, H. H. 2006. Image-based material editing. In Proc. ACM SIGGRAPH, 654–663. Google ScholarDigital Library
23. Kim, V. G., Lipman, Y., and Funkhouser, T. 2012. Symmetry-guided texture synthesis and manipulation. ACM Trans. Graph. 31, 3 (June), 22:1–22:14. Google ScholarDigital Library
24. Kolmogorov, V. 2006. Convergent tree-reweighted message passing for energy minimization. IEEE TPAMI 28, 10 (Oct.), 1568–1583. Google ScholarDigital Library
25. Kopf, J., Neubert, B., Chen, B., Cohen, M., Cohen-Or, D., Deussen, O., Uyttendaele, M., and Lischinski, D. 2008. Deep photo: model-based photograph enhancement and viewing. In Proc. ACM SIGGRAPH Asia, 116:1–116:10. Google ScholarDigital Library
26. Kraevoy, V., Sheffer, A., and Gotsman, C. 2003. Match-maker: constructing constrained texture maps. ACM Trans. Graph. 22, 3 (July), 326–333. Google ScholarDigital Library
27. Kraevoy, V., Sheffer, A., and van de Panne, M. 2009. Modeling from contour drawings. In Proceedings of the 6th Eurographics Symposium on Sketch-Based Interfaces and Modeling, ACM, New York, NY, USA, SBIM ’09, 37–44. Google ScholarDigital Library
28. Kwatra, V., Schodl, A., Essa, I., Turk, G., and Bobick, A. 2003. Graphcut textures: Image and video synthesis using graph cuts. ACM Trans. Graph. 22, 3 (July), 277–286. Google ScholarDigital Library
29. Lalonde, J.-F., Hoiem, D., Efros, A. A., Rother, C., Winn, J., and Criminisi, A. 2007. Photo clip art. Proc. ACM SIGGRAPH 26, 3 (August), 3. Google ScholarDigital Library
30. Land, E. H., John, and Mccann, J. 1971. Lightness and retinex theory. Journal of the Optical Society of America, 1–11.Google ScholarCross Ref
31. Lepetit, V., Moreno-Noguer, F., and Fua, P. 2009. Epnp: An accurate o(n) solution to the pnp problem. Int. J. Comput. Vision 81, 155–166. Google ScholarDigital Library
32. Lim, J., Piriavash, H., and Torralba, A. 2013. Parsing ikea objects: Fine pose estimation. In ICCV. Google ScholarDigital Library
33. Mei, X., Ling, H., and Jacobs, D. 2009. Sparse representation of cast shadows via l1-regularized least squares. In ICCV.Google Scholar
34. Mitra, N. J., and Pauly, M. 2008. Symmetry for architectural design. In Advances in Architectural Geometry, 13–16.Google Scholar
35. Mitra, N. J., Guibas, L. J., and Pauly, M. 2006. Partial and approximate symmetry detection for 3d geometry. In Proc. ACM SIGGRAPH, 560–568. Google ScholarDigital Library
36. Nealen, A., Sorkine, O., Alexa, M., and Cohen-Or, D. 2005. A sketch-based interface for detail-preserving mesh editing. ACM Trans. Graph. 24, 3 (July), 1142–1147. Google ScholarDigital Library
37. Ng, R., Ramamoorthi, R., and Hanrahan, P. 2003. All-frequency shadows using non-linear wavelet lighting approximation. In Proc. ACM SIGGRAPH, 376–381. Google ScholarDigital Library
38. Oh, B. M., Chen, M., Dorsey, J., and Durand, F. 2001. Image-based modeling and photo editing. In Proc. ACM SIGGRAPH, 433–442. Google ScholarDigital Library
39. Okabe, T., Sato, I., and Sato, Y. 2004. Spherical harmonics vs. haar wavelets: Basis for recovering illumination from cast shadows. In CVPR, 50–57.Google Scholar
40. Panagopoulos, A., Samaras, D., and Paragios, N. 2009. Robust shadow and illumination estimation using a mixture model. In CVPR, 651–658.Google Scholar
41. Pauly, M., Mitra, N. J., Giesen, J., Gross, M., and Guibas, L. J. 2005. Example-based 3d scan completion. In Proc. SGP. Google ScholarDigital Library
42. Prasad, M., Zisserman, A., and Fitzgibbon, A. W. 2006. Single view reconstruction of curved surfaces. In CVPR. Google ScholarDigital Library
43. Ramamoorthi, R., and Hanrahan, P. 2001. On the relationship between radiance and irradiance: determining the illumination from images of a convex lambertian object. J. Opt. Soc. Am. A 18, 10, 2448–2459.Google ScholarCross Ref
44. Romeiro, F., and Zickler, T. 2010. Blind reflectometry. In ECCV, 45–58. Google ScholarDigital Library
45. Simpson, J., 2003. Oxford English Dictionary Online, 2nd edition. http://www.oed.com/, July.Google Scholar
46. Sorkine, O., and Alexa, M. 2007. As-rigid-as-possible surface modeling. In Proc. SGP, 109–116. Google ScholarDigital Library
47. Terzopoulos, D., Witkin, A., and Kass, M. 1987. Symmetry-seeking models and 3d object reconstruction. International Journal of Computer Vision 1, 211–221.Google ScholarCross Ref
48. Tzur, Y., and Tal, A. 2009. Flexistickers: photogrammetric texture mapping using casual images. ACM Trans. Graph. 28, 3 (July), 45:1–45:10. Google ScholarDigital Library
49. Xu, K., Zheng, H., Zhang, H., Cohen-Or, D., Liu, L., and Xiong, Y. 2011. Photo-inspired model-driven 3d object modeling. ACM Transactions on Graphics 30, 4. Google ScholarDigital Library
50. Zheng, Y., Chen, X., Cheng, M.-M., Zhou, K., Hu, S.-M., and Mitra, N. J. 2012. Interactive images: cuboid proxies for smart image manipulation. ACM Trans. Graph. 31, 4 (July), 99:1–99:11. Google ScholarDigital Library
51. Zou, H., and Hastie, T. 2005. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society, Series B 67, 301–320.Google ScholarCross Ref
