“3-Sweep: extracting editable objects from a single photo”
Conference:
Type(s):
Title:
- 3-Sweep: extracting editable objects from a single photo
Session/Category Title: Image & Video Editing
Presenter(s)/Author(s):
Abstract:
We introduce an interactive technique for manipulating simple 3D shapes based on extracting them from a single photograph. Such extraction requires understanding of the components of the shape, their projections, and relations. These simple cognitive tasks for humans are particularly difficult for automatic algorithms. Thus, our approach combines the cognitive abilities of humans with the computational accuracy of the machine to solve this problem. Our technique provides the user the means to quickly create editable 3D parts—human assistance implicitly segments a complex object into its components, and positions them in space. In our interface, three strokes are used to generate a 3D component that snaps to the shape’s outline in the photograph, where each stroke defines one dimension of the component. The computer reshapes the component to fit the image of the object in the photograph as well as to satisfy various inferred geometric constraints imposed by its global 3D structure. We show that with this intelligent interactive modeling tool, the daunting task of object extraction is made simple. Once the 3D object has been extracted, it can be quickly edited and placed back into photos or 3D scenes, permitting object-driven photo editing tasks which are impossible to perform in image-space. We show several examples and present a user study illustrating the usefulness of our technique.
References:
1. Andre, A., and Saito, S. 2011. Single-view sketch based modeling. In Proceedings of the Eighth Eurographics Symposium on Sketch-Based Interfaces and Modeling, 133–140.
2. Angelidis, A., Canif, M., Wyvill, G., and King, S. 2004. Swirling-sweepers: Constant-volume modeling. In 12th Pacific Conference on Computer Graphics and Applications, 2004. PG 2004. Proceedings, 10–15.
3. Arbelaez, P., Maire, M., Fowlkes, C., and Malik, J. 2011. Contour detection and hierarchical image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 33, 5, 898–916.
4. Arikan, M., Schwärzler, M., Flöry, S., Wimmer, M., and Maierhofer, S. 2013. O-snap: Optimization-based snapping for modeling architecture. ACM Transactions on Graphics (TOG) 32, 1, 6.
5. Barrett, W., and Cheney, A. 2002. Object-based image editing. In ACM Transactions on Graphics (TOG), vol. 21, 777–784.
6. Benko, P., Kós, G., Várady, T., Andor, L., and Martin, R. 2002. Constrained fitting in reverse engineering. Computer Aided Geometric Design 19, 3, 173–205.
7. Binford, T. O. 1971. Visual perception by computer. In IEEE conference on Systems and Control, vol. 261, 262.
8. Bolle, R. M., and Vemuri, B. C. 1991. On three-dimensional surface reconstruction methods. IEEE Transactions on Pattern Analysis and Machine Intelligence 13, 1, 1–13.
9. Cheng, M., Zhang, F., Mitra, N., Huang, X., and Hu, S. 2010. Repfinder: finding approximately repeated scene elements for image editing. ACM Transactions on Graphics (TOG) 29, 4, 83.
10. Cheng, M. 2009. Curve structure extraction for cartoon images. In Proceedings of The 5th Joint Conference on Harmonious Human Machine Environment, 13–25.
11. Choi, B., and Lee, C. 1990. Sweep surfaces modelling via coordinate transformation and blending. Computer-Aided Design 22, 2, 87–96.
12. Debevec, P. E., Taylor, C. J., and Malik, J. 1996. Modeling and rendering architecture from photographs: A hybrid geometry-and image-based approach. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, 11–20.
13. Eitz, M., Sorkine, O., and Alexa, M. 2007. Sketch based image deformation. In Proceedings of Vision, Modeling and Visualization (VMV), 135–142.
14. Gal, R., Sorkine, O., Mitra, N., and Cohen-Or, D. 2009. iwires: an analyze-and-edit approach to shape manipulation. In ACM Transactions on Graphics (TOG), vol. 28, ACM, 33.
15. Gingold, Y., Igarashi, T., and Zorin, D. 2009. Structured annotations for 2d-to-3d modeling. In ACM Transactions on Graphics (TOG), vol. 28, ACM, 148.
16. Goldberg, C., Chen, T., Zhang, F., Shamir, A., and Hu, S. 2012. Data-driven object manipulation in images. In Computer Graphics Forum, vol. 31, 265–274.
17. Jiang, N., Tan, P., and Cheong, L. 2009. Symmetric architecture modeling with a single image. ACM Transactions on Graphics (TOG) 28, 5, 113.
18. Lalonde, J., Hoiem, D., Efros, A., Rother, C., Winn, J., and Criminisi, A. 2007. Photo clip art. In ACM Transactions on Graphics (TOG), vol. 26, 3.
19. Langbein, F., Marshall, A., and Martin, R. 2004. Choosing consistent constraints for beautification of reverse engineered geometric models. Computer-Aided Design 36, 3, 261–278.
20. Li, Y., Wu, X., Chrysathou, Y., Sharf, A., Cohen-Or, D., and Mitra, N. 2011. Globfit: Consistently fitting primitives by discovering global relations. In ACM Transactions on Graphics (TOG), vol. 30, 52.
21. Merrell, P., and Manocha, D. 2011. Model synthesis: A general procedural modeling algorithm. IEEE Transactions on Visualization and Computer Graphics 17, 6, 715–728.
22. Metaxas, D. N. 1996. Physics-based deformable models: applications to computer vision, graphics, and medical imaging. Kluwer Academic Publishers.
23. Mille, J., Boné, R., and Cohen, L. D. 2008. Region-based 2d deformable generalized cylinder for narrow structures segmentation. In ECCV 2008. 392–404.
24. Oh, B., Chen, M., Dorsey, J., and Durand, F. 2001. Image-based modeling and photo editing. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques, 433–442.
25. Olsen, L., and Samavati, F. F. 2010. Image-assisted modeling from sketches. In Proceedings of Graphics Interface 2010, Canadian Information Processing Society, 225–232.
26. Oswald, M. R., Toppe, E., and Cremers, D. 2012. Fast and globally optimal single view reconstruction of curved objects. In IEEE CVPR, 534–541.
27. Russell, B., and Torralba, A. 2009. Building a database of 3d scenes from user annotations. In IEEE CVPR, 2711–2718.
28. Schmidt, R., and Wyvill, B. 2005. Generalized sweep templates for implicit modeling. In Proceedings of the 3rd international conference on Computer graphics and interactive techniques in Australasia and South East Asia, 187–196.
29. Schmidt, R., Wyvill, B., and Sousa, M. C. 2005. Sketch-based modeling with the blob tree. In ACM SIGGRAPH 2005 Sketches, 90.
30. Schmidt, R., Khan, A., Singh, K., and Kurtenbach, G. 2009. Analytic drawing of 3d scaffolds. In ACM Transactions on Graphics (TOG), vol. 28, 149.
31. Seitz, S., Curless, B., Diebel, J., Scharstein, D., and Szeliski, R. 2006. A comparison and evaluation of multi-view stereo reconstruction algorithms. In IEEE CVPR, vol. 1, 519–528.
32. Shtof, A., Agathos, A., Gingold, Y., Shamir, A., and Cohen-Or, D. 2013. Geosemantic snapping for sketch-based modeling. In Eurographics.
33. Snavely, N. 2011. Scene reconstruction and visualization from internet photo collections: A survey. IPSJ Transactions on Computer Vision and Applications 3, 0, 44–66.
34. Tan, P., Zeng, G., Wang, J., Kang, S. B., and Quan, L. 2007. Image-based tree modeling. ACM Transactions on Graphics (TOG) 26, 3, 87.
35. Terzopoulos, D., Witkin, A., and Kass, M. 1988. Constraints on deformable models: Recovering 3d shape and nonrigid motion. Artificial intelligence 36, 1, 91–123.
36. Tsang, S., Balakrishnan, R., Singh, K., and Ranjan, A. 2004. A suggestive interface for image guided 3d sketching. In Proceedings of the SIGCHI conference on Human factors in computing systems, 591–598.
37. Xiao, C., Liu, M., Yongwei, N., and Dong, Z. 2011. Fast exact nearest patch matching for patch-based image editing and processing. IEEE Transactions on Visualization and Computer Graphics 17, 8, 1122–1134.
38. Xu, K., Zheng, H., Zhang, H., Cohen-Or, D., Liu, L., and Xiong, Y. 2011. Photo-inspired model-driven 3d object modeling. In ACM Transactions on Graphics (TOG), vol. 30, 80.
39. Xu, K., Zhang, H., Cohen-Or, D., and Chen, B. 2012. Fit and diverse: Set evolution for inspiring 3d shape galleries. ACM Transactions on Graphics (TOG) 31, 4, 57.
40. Xue, T., Liu, J., and Tang, X. 2011. Symmetric piecewise planar object reconstruction from a single image. In IEEE CVPR, 2577–2584.
41. Zeleznik, R. C., Herndon, K. P., and Hughes, J. F. 1996. Sketch: an interface for sketching 3d scenes. In ACM SIGGRAPH, 163–170.
42. Zhang, X., Gao, Y., and Caelli, T. 2010. Primitive-based 3d structure inference from a single 2d image for insect modeling: Towards an electronic field guide for insect identification. In International Conference on Control Automation Robotics & Vision (ICARCV), 866–871.
43. Zheng, Y., Fu, H., Cohen-Or, D., Au, O., and Tai, C. 2011. Component-wise controllers for structure-preserving shape manipulation. In Computer Graphics Forum, vol. 30, 563–572.
44. Zheng, Y., Chen, X., Cheng, M., Zhou, K., Hu, S., and Mitra, N. 2012. Interactive images: cuboid proxies for smart image manipulation. ACM Transactions on Graphics (TOG) 31, 4, 99.
45. Zhou, S., Fu, H., Liu, L., Cohen-Or, D., and Han, X. 2010. Parametric reshaping of human bodies in images. ACM Transactions on Graphics (TOG) 29, 4, 126.
