“Interactive sketching of urban procedural models” by Nishida, Garcia-Dorado, Aliaga, Benes and Bousseau

  • ©Gen Nishida, Ignacio Garcia-Dorado, Daniel G. Aliaga, Bedrich Benes, and Adrien Bousseau



Session Title:



    Interactive sketching of urban procedural models




    3D modeling remains a notoriously difficult task for novices despite significant research effort to provide intuitive and automated systems. We tackle this problem by combining the strengths of two popular domains: sketch-based modeling and procedural modeling. On the one hand, sketch-based modeling exploits our ability to draw but requires detailed, unambiguous drawings to achieve complex models. On the other hand, procedural modeling automates the creation of precise and detailed geometry but requires the tedious definition and parameterization of procedural models. Our system uses a collection of simple procedural grammars, called snippets, as building blocks to turn sketches into realistic 3D models. We use a machine learning approach to solve the inverse problem of finding the procedural model that best explains a user sketch. We use non-photorealistic rendering to generate artificial data for training convolutional neural networks capable of quickly recognizing the procedural rule intended by a sketch and estimating its parameters. We integrate our algorithm in a coarse-to-fine urban modeling system that allows users to create rich buildings by successively sketching the building mass, roof, facades, windows, and ornaments. A user study shows that by using our approach non-expert users can generate complex buildings in just a few minutes.


    1. Abdel-Hamid, O., Mohamed, A.-R., Jiang, H., Deng, L., Penn, G., and Yu, D. 2014. Convolutional neural networks for speech recognition. IEEE/ACM Trans. Audio, Speech and Lang. Proc. 22, 10, 1533–1545. Google ScholarDigital Library
    2. Akinlar, C., and Topal, C. 2011. Edlines: Real-time line segment detection by edge drawing (ed). In ICIP, 2837–2840.Google Scholar
    3. Anastacio, F., Prusinkiewicz, P., and Sousa, M. C. 2009. Sketch-based parameterization of l-systems using illustration-inspired construction lines and depth modulation. Comp. & Graph. 33, 4, 440–451. Google ScholarDigital Library
    4. Applegate, C. S., Laycock, S. D., and Day, A. 2012. A sketch-based system for highway design with user-specified regions of influence. Comp. & Graph. 36, 6, 685–695. Google ScholarDigital Library
    5. Bae, S., Balakrishnan, R., and Singh, K. 2008. Ilovesketch: as-natural-as-possible sketching system for creating 3d curve models. In User Interface Software and Technology. Google ScholarDigital Library
    6. Bell, S., and Bala, K. 2015. Learning visual similarity for product design with convolutional neural networks. ACM Trans. Graph. 34, 4, 98:1–98:10. Google ScholarDigital Library
    7. Chen, G., Esch, G., Wonka, P., Müller, P., and Zhang, E. 2008. Interactive procedural street modeling. ACM Trans. Graph. 27, 3, 103:1–103:10. Google ScholarDigital Library
    8. Chen, X., Kang, S. B., Xu, Y.-Q., Dorsey, J., and Shum, H.-Y. 2008. Sketching reality: Realistic interpretation of architectural designs. ACM Trans. Graph. 27, 2, 11:1–11:15. Google ScholarDigital Library
    9. Eigen, D., Puhrsch, C., and Fergus, R. 2014. Depth map prediction from a single image using a multi-scale deep network. In Neural Information Processing Systems (NIPS). Google ScholarDigital Library
    10. Eissen, K., and Steur, R. 2009. Sketching: Drawing Techniques for Product Designers. BIS Publishers.Google Scholar
    11. Eitz, M., Richter, R., Boubekeur, T., Hildebrand, K., and Alexa, M. 2012. Sketch-based shape retrieval. ACM Trans. Graph. 31, 4, 31:1–31:10. Google ScholarDigital Library
    12. Emilien, A., Vimont, U., Cani, M.-P., Poulin, P., and Benes, B. 2015. Worldbrush: Interactive example-based synthesis of procedural virtual worlds. ACM Trans. Graph. 34, 4, 106:1–106:11. Google ScholarDigital Library
    13. Igarashi, T., Matsuoka, S., and Tanaka, H. 1999. Teddy: a sketching interface for 3d freeform design. In Proc. of Siggraph, 409–416. Google ScholarDigital Library
    14. Ijiri, T., Owada, S., and Igarashi, T. 2006. The sketch l-system: Global control of tree modeling using free-form strokes. In Smart Graphics, Springer, 138–146.Google ScholarCross Ref
    15. Jampani, V., Nowozin, S., Loper, M., and Gehler, P. V. 2015. The informed sampler: A discriminative approach to bayesian inference in generative computer vision models. Computer Vision and Image Understanding 136, 32–44. Google ScholarDigital Library
    16. Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., and Darrell, T. 2014. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093.Google Scholar
    17. Krizhevsky, A., Sutskever, I., and Hinton, G. E. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, 1097–1105.Google Scholar
    18. Lecun, Y., Bottou, L., Bengio, Y., and Haffner, P. 1998. Gradient-based learning applied to document recognition. Proceedings of the IEEE 86, 11, 2278–2324.Google ScholarCross Ref
    19. Lipp, M., Wonka, P., and Wimmer, M. 2008. Interactive visual editing of grammars for procedural architecture. ACM Trans. Graph. 27, 3, 102:1–102:10. Google ScholarDigital Library
    20. Lipson, H., and Shpitalni, M. 1996. Optimization-based reconstruction of a 3d object from a single freehand line drawing. Computer-Aided Design 28, 651–663.Google ScholarCross Ref
    21. Lipson, H., and Shpitalni, M. 2000. Conceptual design and analysis by sketching. Artif. Intell. Eng. Des. Anal. Manuf. 14, 5, 391–401. Google ScholarDigital Library
    22. Longay, S., Runions, A., Boudon, F., and Prusinkiewicz, P. 2012. Treesketch: Interactive procedural modeling of trees on a tablet. In SBIM, Eurographics Association, 107–120. Google ScholarDigital Library
    23. Loomis, A., and Ross, A. 2014. I’d Love to Draw. Titan Books.Google Scholar
    24. Müller, P., Wonka, P., Haegler, S., Ulmer, A., and Van Gool, L. 2006. Procedural modeling of buildings. ACM Trans. Graph. 25, 3, 614–623. Google ScholarDigital Library
    25. Olsen, L., Samavati, F. F., Sousa, M. C., and Jorge, J. A. 2009. Sketch-based modeling: A survey. Comp. & Graph. 33, 1, 85–103. Google ScholarDigital Library
    26. Parish, Y. I., and Müller, P. 2001. Procedural modeling of cities. In Comp. graphics and interactive techniques, ACM, 301–308. Google ScholarDigital Library
    27. Pfister, T., Charles, J., and Zisserman, A. 2015. Flowing convnets for human pose estimation in videos. In ICCV. Google ScholarDigital Library
    28. Prusinkiewicz, P., and Lindenmayer, A. 2012. The algorithmic beauty of plants. Springer Science & Business Media.Google Scholar
    29. Ritchie, D., Mildenhall, B., Goodman, N. D., and Hanrahan, P. 2015. Controlling procedural modeling programs with stochastically-ordered sequential monte carlo. ACM Trans. Graph. 34, 4, 105:1–105:11. Google ScholarDigital Library
    30. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L. 2015. ImageNet Large Scale Visual Recognition Challenge. IJCV 115, 3, 211–252. Google ScholarDigital Library
    31. Schmidt, R., Khan, A., Singh, K., and Kurtenbach, G. 2009. Analytic drawing of 3d scaffolds. ACM Trans. Graph. 28, 5, 149:1–149:10. Google ScholarDigital Library
    32. Shtof, A., Agathos, A., Gingold, Y., Shamir, A., and Cohen-Or, D. 2013. Geosemantic snapping for sketch-based modeling. Comp. Graph. Forum 32, 2, 245–253.Google ScholarCross Ref
    33. Smelik, R., Tutenel, T., de Kraker, K. J., and Bidarra, R. 2010. Interactive creation of virtual worlds using procedural sketching. In Proceedings of eurographics.Google Scholar
    34. Smelik, R. M., Tutenel, T., Bidarra, R., and Benes, B. 2014. A survey on procedural modelling for virtual worlds. In Comp. Graph. Forum, vol. 33, 31–50.Google ScholarDigital Library
    35. Smith, A. R. 1984. Plants, fractals, and formal languages. SIGGRAPH Comput. Graph. 18, 3, 1–10. Google ScholarDigital Library
    36. Stava, O., Pirk, S., Kratt, J., Chen, B., Mch, R., Deussen, O., and Benes, B. 2014. Inverse procedural modelling of trees. Comp. Graph. Forum 33, 6, 118–131.Google ScholarDigital Library
    37. Su, H., Qi, C. R., Li, Y., and Guibas, L. J. 2015. Render for cnn: Viewpoint estimation in images using cnns trained with rendered 3d model views. In ICCV. Google ScholarDigital Library
    38. Talton, J. O., Lou, Y., Lesser, S., Duke, J., Měch, R., and Koltun, V. 2011. Metropolis procedural modeling. ACM Trans. Graph. 30, 2, 11:1–11:14. Google ScholarDigital Library
    39. Vanegas, C. A., Aliaga, D. G., Wonka, P., Müller, P., Waddell, P., and Watson, B. 2010. Modelling the appearance and behaviour of urban spaces. Comp. Graph. Forum 29, 1, 25–42.Google ScholarCross Ref
    40. Vanegas, C. A., Garcia-Dorado, I., Aliaga, D. G., Benes, B., and Waddell, P. 2012. Inverse design of urban procedural models. ACM Trans. Graph. 31, 6, 168:1–168:11. Google ScholarDigital Library
    41. Wang, F., Kang, L., and Li, Y. 2015. Sketch-based 3d shape retrieval using convolutional neural networks. arXiv preprint arXiv:1504.03504.Google Scholar
    42. Wonka, P., Wimmer, M., Sillion, F., and Ribarsky, W. 2003. Instant architecture. ACM Trans. Graph. 22, 3, 669–677. Google ScholarDigital Library
    43. Xie, X., Xu, K., Mitra, N. J., Cohen-Or, D., Gong, W., Su, Q., and Chen, B. 2013. Sketch-to-design: Context-based part assembly. Comp. Graph. Forum 32, 8, 233–245.Google ScholarCross Ref
    44. Xu, B., Chang, W., Sheffer, A., Bousseau, A., Mccrae, J., and Singh, K. 2014. True2form: 3d curve networks from 2d sketches via selective regularization. ACM Trans. on Graph. 33, 4, 131:1–131:13. Google ScholarDigital Library
    45. Xue, T., Liu, J., and Tang, X. 2012. Example-based 3d object reconstruction from line drawings. In CVPR, IEEE, 302–309. Google ScholarDigital Library
    46. Zeleznik, R. C., Herndon, K. P., and Hughes, J. F. 1996. Sketch: An interface for sketching 3d scenes. In Computer Graphics, Proceedings of Siggraph 1996, 163–170. Google ScholarDigital Library
    47. Zheng, Y., Liu, H., Dorsey, J., and Mitra, N. J. 2016. Smartcanvas; context-inferred interpretation of sketches for preparatory design studies. Comp. Graph. Forum.Google Scholar

ACM Digital Library Publication: