“Sketch2Scene: sketch-based co-retrieval and co-placement of 3D models” by Xu, Chen, Fu, Sun and Hu

  • ©Kun Xu, Kang Chen, Hongbo Fu, Wei-Lun Sun, and Shi-Min Hu



Session Title:

    Structures, Faces & Building


    Sketch2Scene: sketch-based co-retrieval and co-placement of 3D models




    This work presents Sketch2Scene, a framework that automatically turns a freehand sketch drawing inferring multiple scene objects to semantically valid, well arranged scenes of 3D models. Unlike the existing works on sketch-based search and composition of 3D models, which typically process individual sketched objects one by one, our technique performs co-retrieval and co-placement of 3D relevant models by jointly processing the sketched objects. This is enabled by summarizing functional and spatial relationships among models in a large collection of 3D scenes as structural groups. Our technique greatly reduces the amount of user intervention needed for sketch-based modeling of 3D scenes and fits well into the traditional production pipeline involving concept design followed by 3D modeling. A pilot study indicates that it is promising to use our technique as an alternative but more efficient tool of standard 3D modeling for 3D scene construction.


    1. Belongie, S., Malik, J., and Puzicha, J. 2002. Shape matching and object recognition using shape contexts. IEEE Transactions on Pattern Analysis and Machine Intelligence 24, 4 (Apr.), 509–522. Google ScholarDigital Library
    2. Chaudhuri, S., and Koltun, V. 2010. Data-driven suggestions for creativity support in 3d modeling. ACM Transactions on Graphics 29, 6, 183:1–183:9. Google ScholarDigital Library
    3. Chaudhuri, S., Kalogerakis, E., Guibas, L., and Koltun, V. 2011. Probabilistic reasoning for assembly-based 3d modeling. ACM Transactions on Graphics 30, 4 (Aug.), 35:1–35:10. Google ScholarDigital Library
    4. Chen, T., Cheng, M.-M., Tan, P., Shamir, A., and Hu, S.-M. 2009. Sketch2photo: internet image montage. ACM Transactions on Graphics 28, 5, 124:1–124:10. Google ScholarDigital Library
    5. Cheng, M.-M., Zhang, F.-L., Mitra, N. J., Huang, X., and Hu, S.-M. 2010. Repfinder: finding approximately repeated scene elements for image editing. ACM Transactions on Graphics 29, 4 (July), 83:1–83:8. Google ScholarDigital Library
    6. Chopine, A. 2011. 3D Art Essentials: The Fundamentals of 3D Modeling, Texturing, and Animation. Focal Press. Google ScholarDigital Library
    7. DeCarlo, D., Finkelstein, A., Rusinkiewicz, S., and Santella, A. 2003. Suggestive contours for conveying shape. ACM Transactions on Graphics 22, 3 (July), 848–855. Google ScholarDigital Library
    8. Divvala, S. K., Hoiem, D., Hays, J. H., Efros, A. A., and Hebert, M. 2009. An empirical study of context in object detection. In Proceedings of CVPR, 1271–1278.Google Scholar
    9. Dutagaci, H., Cheung, C. P., and Godil, A. 2010. A benchmark for best view selection of 3d objects. In Proceedings of the ACM workshop on 3D object retrieval, ACM, New York, NY, USA, 45–50. Google ScholarDigital Library
    10. Eitz, M., Richter, R., Boubekeur, T., Hildebrand, K., and Alexa, M. 2012. Sketch-based shape retrieval. ACM Transactions on Graphics 31, 4, 31:1–31:10. Google ScholarDigital Library
    11. Fisher, M., and Hanrahan, P. 2010. Context-based search for 3d models. ACM Transactions on Graphics 29, 6 (Dec.), 182:1–182:10. Google ScholarDigital Library
    12. Fisher, M., Savva, M., and Hanrahan, P. 2011. Characterizing structural relationships in scenes using graph kernels. ACM Transactions on Graphics 30, 4 (July), 34:1–34:12. Google ScholarDigital Library
    13. Fisher, M., Ritchie, D., Savva, M., Funkhouser, T., and Hanrahan, P. 2012. Example-based synthesis of 3d object arrangements. ACM Transactions on Graphics 31, 6 (Nov.), 135:1–135:11. Google ScholarDigital Library
    14. Funkhouser, T., Kazhdan, M., Shilane, P., Min, P., Kiefer, W., Tal, A., Rusinkiewicz, S., and Dobkin, D. 2004. Modeling by example. ACM Transactions on Graphics 23, 3, 652–663. Google ScholarDigital Library
    15. Galleguillos, C., and Belongie, S. 2010. Context based object categorization: A critical survey. Computer Vision and Image Understanding 114, 6, 712–722. Google ScholarDigital Library
    16. Hays, J., and Efros, A. A. 2007. Scene completion using millions of photographs. ACM Transactions on Graphics 26, 3, 4:1–4:7. Google ScholarDigital Library
    17. Huang, Q., Koltun, V., and Guibas, L. 2011. Joint shape segmentation with linear programming. ACM Transactions on Graphics 30, 6, 125:1–125:11. Google ScholarDigital Library
    18. Johnson, M., Brostow, G., Shotton, J., Arandjelovic, O., Kwatra, V., and Cipolla, R. 2006. Semantic photo synthesis. Computer Graphics Forum 25, 3, 407–413.Google ScholarCross Ref
    19. Kalogerakis, E., Chaudhuri, S., Koller, D., and Koltun, V. 2012. A probabilistic model for component-based shape synthesis. ACM Transactions on Graphics 31, 4, 55:1–55:13. Google ScholarDigital Library
    20. Lalonde, J., Hoiem, D., Efros, A., Rother, C., Winn, J., and Criminisi, A. 2007. Photo clip art. ACM Transactions on Graphics 26, 3 (Aug.), 3:1–3:10. Google ScholarDigital Library
    21. Lee, J., and Funkhouser, T. 2008. Sketch-based search and composition of 3D models. In Proceedings of EUROGRAPHICS Workshop on Sketch-Based Interfaces and Modeling, 97–104. Google ScholarDigital Library
    22. Lee, Y. J., Zitnick, C. L., and Cohen, M. F. 2011. Shadowdraw: real-time user guidance for freehand drawing. ACM Transactions on Graphics 30, 27:1–27:10. Google ScholarDigital Library
    23. Liu, H., Zhang, L., and Huang, H. 2012. Web-image driven best views of 3d shapes. the Visual Computer 28, 3, 279–287. Google ScholarDigital Library
    24. Merrell, P., Schkufza, E., Li, Z., Agrawala, M., and Koltun, V. 2011. Interactive furniture layout using interior design guidelines. ACM Transactions on Graphics 30, 4 (July), 87:1–87:10. Google ScholarDigital Library
    25. Noris, G., Sýkora, D., Shamir, A., Coros, S., Whited, B., Simmons, M., Hornung, A., Gross, M., and Sumner, R. 2012. Smart scribbles for sketch segmentation. Computer Graphics Forum 31, 8, 2516–2527. Google ScholarDigital Library
    26. Olsen, L., Samavati, F., Sousa, M., and Jorge, J. 2009. Sketch-based modeling: A survey. Computers & Graphics 33, 1, 85–103. Google ScholarDigital Library
    27. Russell, B., and Torralba, A. 2009. Building a database of 3d scenes from user annotations. In Proceedings of CVPR, 2711–2718.Google Scholar
    28. Russell, B. C., Torralba, A., Murphy, K. P., and Freeman, W. T. 2008. Labelme: a database and web-based tool for image annotation. International journal of computer vision 77, 1-3, 157–173. Google ScholarDigital Library
    29. Secord, A., Lu, J., Finkelstein, A., Singh, M., and Nealen, A. 2011. Perceptual models of viewpoint preference. ACM Trans. Graph. 30, 5 (Oct.), 109:1–109:12. Google ScholarDigital Library
    30. Shen, C.-H., Fu, H., Chen, K., and Hu, S.-M. 2012. Structure recovery by part assembly. ACM Transactions on Graphics 31, 6, 180:1–180:11. Google ScholarDigital Library
    31. Shin, H., and Igarashi, T. 2007. Magic canvas: interactive design of a 3-d scene prototype from freehand sketches. In Proceedings of Graphics Interface, 63–70. Google ScholarDigital Library
    32. Sidi, O., van Kaick, O., Kleiman, Y., Zhang, H., and Cohen-Or, D. 2011. Unsupervised co-segmentation of a set of shapes via descriptor-space spectral clustering. ACM Transactions on Graphics 30, 126:1–126:9. Google ScholarDigital Library
    33. Wang, Y., Asafi, S., van Kaick, O., Zhang, H., Cohen-Or, D., and Chen, B. 2012. Active co-analysis of a set of shapes. ACM Transactions on Graphics 31, 6 (Nov.), 165:1–165:10. Google ScholarDigital Library
    34. Xie, X., Xu, K., Mitra, N. J., Cohen-Or, D., and Chen, B. 2012. Sketch-to-design: Context-based part assembly. ArXiv e-prints (Dec.).Google Scholar
    35. Xu, K., Zhang, H., Cohen-Or, D., and Chen, B. 2012. Fit and diverse: set evolution for inspiring 3d shape galleries. ACM Transactions on Graphics 31, 4, 57:1–57:10. Google ScholarDigital Library
    36. Yu, L.-F., Yeung, S.-K., Tang, C.-K., Terzopoulos, D., Chan, T. F., and Osher, S. J. 2011. Make it home: automatic optimization of furniture arrangement. ACM Transactions on Graphics 30, 4 (July), 86:1–86:12. Google ScholarDigital Library
    37. Yumer, M., and Kara, L. 2012. Co-abstraction of shape collections. ACM Transactions on Graphics 31, 6, 166:1–166:11. Google ScholarDigital Library
    38. Zheng, Y., Fu, H., Cohen-Or, D., Au, O. K.-C., and Tai, C.-L. 2011. Component-wise controllers for structure-preserving shape manipulation. Computer Graphics Forum 30, 2, 563–572.Google ScholarCross Ref
    39. Zhu, X., Jin, X., Liu, S., and Zhao, H. 2012. Analytical solutions for sketch-based convolution surface modeling on the gpu. the Visual Computer 28, 11, 1115–1125.Google Scholar

ACM Digital Library Publication: