“Interactive sketching of urban procedural models” by Nishida, Garcia-Dorado, Aliaga, Benes and Bousseau
Conference:
Type:
Title:
- Interactive sketching of urban procedural models
Session/Category Title: PROCEDURAL MODELING
Presenter(s)/Author(s):
Moderator(s):
Abstract:
3D modeling remains a notoriously difficult task for novices despite significant research effort to provide intuitive and automated systems. We tackle this problem by combining the strengths of two popular domains: sketch-based modeling and procedural modeling. On the one hand, sketch-based modeling exploits our ability to draw but requires detailed, unambiguous drawings to achieve complex models. On the other hand, procedural modeling automates the creation of precise and detailed geometry but requires the tedious definition and parameterization of procedural models. Our system uses a collection of simple procedural grammars, called snippets, as building blocks to turn sketches into realistic 3D models. We use a machine learning approach to solve the inverse problem of finding the procedural model that best explains a user sketch. We use non-photorealistic rendering to generate artificial data for training convolutional neural networks capable of quickly recognizing the procedural rule intended by a sketch and estimating its parameters. We integrate our algorithm in a coarse-to-fine urban modeling system that allows users to create rich buildings by successively sketching the building mass, roof, facades, windows, and ornaments. A user study shows that by using our approach non-expert users can generate complex buildings in just a few minutes.
References:
1. Abdel-Hamid, O., Mohamed, A.-R., Jiang, H., Deng, L., Penn, G., and Yu, D. 2014. Convolutional neural networks for speech recognition. IEEE/ACM Trans. Audio, Speech and Lang. Proc. 22, 10, 1533–1545. Google ScholarDigital Library
2. Akinlar, C., and Topal, C. 2011. Edlines: Real-time line segment detection by edge drawing (ed). In ICIP, 2837–2840.Google Scholar
3. Anastacio, F., Prusinkiewicz, P., and Sousa, M. C. 2009. Sketch-based parameterization of l-systems using illustration-inspired construction lines and depth modulation. Comp. & Graph. 33, 4, 440–451. Google ScholarDigital Library
4. Applegate, C. S., Laycock, S. D., and Day, A. 2012. A sketch-based system for highway design with user-specified regions of influence. Comp. & Graph. 36, 6, 685–695. Google ScholarDigital Library
5. Bae, S., Balakrishnan, R., and Singh, K. 2008. Ilovesketch: as-natural-as-possible sketching system for creating 3d curve models. In User Interface Software and Technology. Google ScholarDigital Library
6. Bell, S., and Bala, K. 2015. Learning visual similarity for product design with convolutional neural networks. ACM Trans. Graph. 34, 4, 98:1–98:10. Google ScholarDigital Library
7. Chen, G., Esch, G., Wonka, P., Müller, P., and Zhang, E. 2008. Interactive procedural street modeling. ACM Trans. Graph. 27, 3, 103:1–103:10. Google ScholarDigital Library
8. Chen, X., Kang, S. B., Xu, Y.-Q., Dorsey, J., and Shum, H.-Y. 2008. Sketching reality: Realistic interpretation of architectural designs. ACM Trans. Graph. 27, 2, 11:1–11:15. Google ScholarDigital Library
9. Eigen, D., Puhrsch, C., and Fergus, R. 2014. Depth map prediction from a single image using a multi-scale deep network. In Neural Information Processing Systems (NIPS). Google ScholarDigital Library
10. Eissen, K., and Steur, R. 2009. Sketching: Drawing Techniques for Product Designers. BIS Publishers.Google Scholar
11. Eitz, M., Richter, R., Boubekeur, T., Hildebrand, K., and Alexa, M. 2012. Sketch-based shape retrieval. ACM Trans. Graph. 31, 4, 31:1–31:10. Google ScholarDigital Library
12. Emilien, A., Vimont, U., Cani, M.-P., Poulin, P., and Benes, B. 2015. Worldbrush: Interactive example-based synthesis of procedural virtual worlds. ACM Trans. Graph. 34, 4, 106:1–106:11. Google ScholarDigital Library
13. Igarashi, T., Matsuoka, S., and Tanaka, H. 1999. Teddy: a sketching interface for 3d freeform design. In Proc. of Siggraph, 409–416. Google ScholarDigital Library
14. Ijiri, T., Owada, S., and Igarashi, T. 2006. The sketch l-system: Global control of tree modeling using free-form strokes. In Smart Graphics, Springer, 138–146.Google ScholarCross Ref
15. Jampani, V., Nowozin, S., Loper, M., and Gehler, P. V. 2015. The informed sampler: A discriminative approach to bayesian inference in generative computer vision models. Computer Vision and Image Understanding 136, 32–44. Google ScholarDigital Library
16. Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., and Darrell, T. 2014. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093.Google Scholar
17. Krizhevsky, A., Sutskever, I., and Hinton, G. E. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, 1097–1105.Google Scholar
18. Lecun, Y., Bottou, L., Bengio, Y., and Haffner, P. 1998. Gradient-based learning applied to document recognition. Proceedings of the IEEE 86, 11, 2278–2324.Google ScholarCross Ref
19. Lipp, M., Wonka, P., and Wimmer, M. 2008. Interactive visual editing of grammars for procedural architecture. ACM Trans. Graph. 27, 3, 102:1–102:10. Google ScholarDigital Library
20. Lipson, H., and Shpitalni, M. 1996. Optimization-based reconstruction of a 3d object from a single freehand line drawing. Computer-Aided Design 28, 651–663.Google ScholarCross Ref
21. Lipson, H., and Shpitalni, M. 2000. Conceptual design and analysis by sketching. Artif. Intell. Eng. Des. Anal. Manuf. 14, 5, 391–401. Google ScholarDigital Library
22. Longay, S., Runions, A., Boudon, F., and Prusinkiewicz, P. 2012. Treesketch: Interactive procedural modeling of trees on a tablet. In SBIM, Eurographics Association, 107–120. Google ScholarDigital Library
23. Loomis, A., and Ross, A. 2014. I’d Love to Draw. Titan Books.Google Scholar
24. Müller, P., Wonka, P., Haegler, S., Ulmer, A., and Van Gool, L. 2006. Procedural modeling of buildings. ACM Trans. Graph. 25, 3, 614–623. Google ScholarDigital Library
25. Olsen, L., Samavati, F. F., Sousa, M. C., and Jorge, J. A. 2009. Sketch-based modeling: A survey. Comp. & Graph. 33, 1, 85–103. Google ScholarDigital Library
26. Parish, Y. I., and Müller, P. 2001. Procedural modeling of cities. In Comp. graphics and interactive techniques, ACM, 301–308. Google ScholarDigital Library
27. Pfister, T., Charles, J., and Zisserman, A. 2015. Flowing convnets for human pose estimation in videos. In ICCV. Google ScholarDigital Library
28. Prusinkiewicz, P., and Lindenmayer, A. 2012. The algorithmic beauty of plants. Springer Science & Business Media.Google Scholar
29. Ritchie, D., Mildenhall, B., Goodman, N. D., and Hanrahan, P. 2015. Controlling procedural modeling programs with stochastically-ordered sequential monte carlo. ACM Trans. Graph. 34, 4, 105:1–105:11. Google ScholarDigital Library
30. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L. 2015. ImageNet Large Scale Visual Recognition Challenge. IJCV 115, 3, 211–252. Google ScholarDigital Library
31. Schmidt, R., Khan, A., Singh, K., and Kurtenbach, G. 2009. Analytic drawing of 3d scaffolds. ACM Trans. Graph. 28, 5, 149:1–149:10. Google ScholarDigital Library
32. Shtof, A., Agathos, A., Gingold, Y., Shamir, A., and Cohen-Or, D. 2013. Geosemantic snapping for sketch-based modeling. Comp. Graph. Forum 32, 2, 245–253.Google ScholarCross Ref
33. Smelik, R., Tutenel, T., de Kraker, K. J., and Bidarra, R. 2010. Interactive creation of virtual worlds using procedural sketching. In Proceedings of eurographics.Google Scholar
34. Smelik, R. M., Tutenel, T., Bidarra, R., and Benes, B. 2014. A survey on procedural modelling for virtual worlds. In Comp. Graph. Forum, vol. 33, 31–50.Google ScholarDigital Library
35. Smith, A. R. 1984. Plants, fractals, and formal languages. SIGGRAPH Comput. Graph. 18, 3, 1–10. Google ScholarDigital Library
36. Stava, O., Pirk, S., Kratt, J., Chen, B., Mch, R., Deussen, O., and Benes, B. 2014. Inverse procedural modelling of trees. Comp. Graph. Forum 33, 6, 118–131.Google ScholarDigital Library
37. Su, H., Qi, C. R., Li, Y., and Guibas, L. J. 2015. Render for cnn: Viewpoint estimation in images using cnns trained with rendered 3d model views. In ICCV. Google ScholarDigital Library
38. Talton, J. O., Lou, Y., Lesser, S., Duke, J., Měch, R., and Koltun, V. 2011. Metropolis procedural modeling. ACM Trans. Graph. 30, 2, 11:1–11:14. Google ScholarDigital Library
39. Vanegas, C. A., Aliaga, D. G., Wonka, P., Müller, P., Waddell, P., and Watson, B. 2010. Modelling the appearance and behaviour of urban spaces. Comp. Graph. Forum 29, 1, 25–42.Google ScholarCross Ref
40. Vanegas, C. A., Garcia-Dorado, I., Aliaga, D. G., Benes, B., and Waddell, P. 2012. Inverse design of urban procedural models. ACM Trans. Graph. 31, 6, 168:1–168:11. Google ScholarDigital Library
41. Wang, F., Kang, L., and Li, Y. 2015. Sketch-based 3d shape retrieval using convolutional neural networks. arXiv preprint arXiv:1504.03504.Google Scholar
42. Wonka, P., Wimmer, M., Sillion, F., and Ribarsky, W. 2003. Instant architecture. ACM Trans. Graph. 22, 3, 669–677. Google ScholarDigital Library
43. Xie, X., Xu, K., Mitra, N. J., Cohen-Or, D., Gong, W., Su, Q., and Chen, B. 2013. Sketch-to-design: Context-based part assembly. Comp. Graph. Forum 32, 8, 233–245.Google ScholarCross Ref
44. Xu, B., Chang, W., Sheffer, A., Bousseau, A., Mccrae, J., and Singh, K. 2014. True2form: 3d curve networks from 2d sketches via selective regularization. ACM Trans. on Graph. 33, 4, 131:1–131:13. Google ScholarDigital Library
45. Xue, T., Liu, J., and Tang, X. 2012. Example-based 3d object reconstruction from line drawings. In CVPR, IEEE, 302–309. Google ScholarDigital Library
46. Zeleznik, R. C., Herndon, K. P., and Hughes, J. F. 1996. Sketch: An interface for sketching 3d scenes. In Computer Graphics, Proceedings of Siggraph 1996, 163–170. Google ScholarDigital Library
47. Zheng, Y., Liu, H., Dorsey, J., and Mitra, N. J. 2016. Smartcanvas; context-inferred interpretation of sketches for preparatory design studies. Comp. Graph. Forum.Google Scholar
