“FrankenGAN: guided detail synthesis for building mass models using style-synchonized GANs”
Conference:
Type(s):
Title:
- FrankenGAN: guided detail synthesis for building mass models using style-synchonized GANs
Session/Category Title: Geometry generation
Presenter(s)/Author(s):
Moderator(s):
Abstract:
Coarse building mass models are now routinely generated at scales ranging from individual buildings to whole cities. Such models can be abstracted from raw measurements, generated procedurally, or created manually. However, these models typically lack any meaningful geometric or texture details, making them unsuitable for direct display. We introduce the problem of automatically and realistically decorating such models by adding semantically consistent geometric details and textures. Building on the recent success of generative adversarial networks (GANs), we propose FrankenGAN, a cascade of GANs that creates plausible details across multiple scales over large neighborhoods. The various GANs are synchronized to produce consistent style distributions over buildings and neighborhoods. We provide the user with direct control over the variability of the output. We allow him/her to interactively specify the style via images and manipulate style-adapted sliders to control style variability. We test our system on several large-scale examples. The generated outputs are qualitatively evaluated via a set of perceptual studies and are found to be realistic, semantically plausible, and consistent in style.
References:
1. Martin Arjovsky, Soumith Chintala, and Léon Bottou. 2017. Wasserstein GAN. (2017). arXiv:1701.07875Google Scholar
2. Fan Bao, Michael Schwarz, and Peter Wonka. 2013. Procedural Facade Variations from a Single Layout. ACM TOG 32, 1 (2013), 8. Google ScholarDigital Library
3. M. Berger, J. Li, and J. A. Levine. 2018. A Generative Model for Volume Rendering. IEEE TVCG (2018), 1–1.Google Scholar
4. David Berthelot, Tom Schumm, and Luke Metz. 2017. BEGAN: Boundary Equilibrium Generative Adversarial Networks. (2017). arXiv:1703.10717Google Scholar
5. Martin Bokeloh, Michael Wand, Hans-Peter Seidel, and Vladlen Koltun. 2012. An Algebraic Model for Parameterized Shape Editing. ACM TOG 31, 4 (2012). Google ScholarDigital Library
6. A. Cohen, A. G. Schwing, and M. Pollefeys. 2014. Efficient Structured Parsing of Facades Using Dynamic Programming. In IEEE PAMI. 3206–3213. Google ScholarDigital Library
7. Minh Dang, Duygu Ceylan, Boris Neubert, and Mark Pauly. 2014. SAFE: Structure-aware Facade Editing. CGF 33, 2 (2014), 83–93. Google ScholarDigital Library
8. Emily L Denton, Soumith Chintala, Arthur Szlam, and Rob Fergus. 2015. Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks. In NIPS. 1486–1494. Google ScholarDigital Library
9. Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. 2016. Adversarial Feature Learning. In ICLR.Google Scholar
10. Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Alex Lamb, Martin Arjovsky, Olivier Mastropietro, and Aaron Courville. 2016. Adversarially Learned Inference. In ICLR.Google Scholar
11. Victor Escorcia, Fabian Caba Heilbron, Juan Carlos Niebles, and Bernard Ghanem. 2016. Daps: Deep action proposals for action understanding. In ECCV. Springer, 768–784.Google Scholar
12. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative Adversarial Nets. In NIPS. 2672–2680. Google ScholarDigital Library
13. Swaminathan Gurumurthy, Ravi Kiran Sarvadevabhatla, and R. Venkatesh Babu. 2017. DeLiGAN: Generative Adversarial Networks for Diverse and Limited Data. In CVPR.Google Scholar
14. Stefan Hartmann, Michael Weinmann, Raoul Wessel, and Reinhard Klein. 2017. Street-GAN: Towards Road Network Synthesis with Generative Adversarial Networks. ICCV.Google Scholar
15. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR. 770–778.Google Scholar
16. Martin Ilčík, Przemyslaw Musialski, Thomas Auzinger, and Michael Wimmer. 2015. Layer-Based Procedural Design of Façades. CGF 34, 2 (2015), 205–216. Google ScholarDigital Library
17. Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017. Image-to-Image Translation with Conditional Adversarial Networks. CVPR (2017).Google Scholar
18. Justin Johnson, Alexandre Alahi, and Fei fei Li. 2016. Perceptual Losses for Real-Time Style Transfer and Super-Resolution. In ECCV.Google Scholar
19. Angel X. Chang Kai Wang, Manolis Savva and Daniel Ritchie. 2018. Deep Convolutional Priors for Indoor Scene Synthesis. ACM TOG (2018). Google ScholarDigital Library
20. Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. 2018. Progressive Growing of GANs for Improved Quality, Stability, and Variation. In ICLR.Google Scholar
21. Tom Kelly, John Femiani, Peter Wonka, and Niloy J. Mitra. 2017. BigSUR: Large-scale Structured Urban Reconstruction. ACM SIGGRAPH Asia 36, 6, Article 204 (2017). Google ScholarDigital Library
22. D.P. Kingma and M. Welling. 2014. Auto-Encoding Variational Bayes. In ICLR.Google Scholar
23. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In ICLR.Google Scholar
24. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In NIPS. 1097–1105. Google ScholarDigital Library
25. Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole Winther. 2016. Autoencoding beyond pixels using a learned similarity metric. In ICML, Vol. 48. Google ScholarDigital Library
26. Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi. 2017. Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. In CVPR.Google Scholar
27. Jinjie Lin, Daniel Cohen-Or, Hao Zhang, Cheng Liang, Andrei Sharf, Oliver Deussen, and Baoquan Chen. 2011. Structure-Preserving Retargeting of Irregular 3D Architecture. ACM TOG 30, 6 (2011), 183:1–183:10. Google ScholarDigital Library
28. Jonathan Long, Evan Shelhamer, and Trevor Darrell. 2015. Fully convolutional networks for semantic segmentation. In CVPR. 3431–3440.Google Scholar
29. Andjelo Martinovic and Luc Van Gool. 2013. Bayesian Grammar Learning for Inverse Procedural Modeling. In CVPR. 201–208. Google ScholarDigital Library
30. Markus Mathias, Andelo Martinović, and Luc Gool. 2016. ATLAS: A Three-Layered Approach to Facade Parsing. IJCV 118, 1 (May 2016), 22–48. Google ScholarDigital Library
31. Pascal Mueller, Peter Wonka, Simon Haegler, Andreas Ulmer, and Luc Van Gool. 2006. Procedural Modeling of Buildings. ACM TOG 25, 3 (2006), 614–623. Google ScholarDigital Library
32. Gen Nishida, Ignacio Garcia-Dorado, Daniel G. Aliaga, Bedrich Benes, and Adrien Bousseau. 2016. Interactive Sketching of Urban Procedural Models. ACM TOG (2016). Google ScholarDigital Library
33. Chi-Han Peng, Yong-Liang Yang, Fan Bao, Daniel Fink, Dong-Ming Yan, Peter Wonka, and Niloy J Mitra. 2016. Computational Network Design from Functional Specifications. ACM TOG 35, 4 (2016), 131. Google ScholarDigital Library
34. Daniel Ritchie, Ben Mildenhall, Noah D. Goodman, and Pat Hanrahan. 2015. Controlling Procedural Modeling Programs with Stochastically-Ordered Sequential Monte Carlo. ACM TOG 34, 4 (2015), 105:1–105:11. Google ScholarDigital Library
35. Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, and Xi Chen. 2016. Improved Techniques for Training GANs. In NIPS. 2234–2242. Google ScholarDigital Library
36. Michael Schwarz and Pascal Müller. 2015. Advanced Procedural Modeling of Architecture. ACM TOG 34, 4 (2015), 107:1–107:12. Google ScholarDigital Library
37. Jost T. Springenberg. 2016. Unsupervised and Semi-supervised Learning with Categorical Generative Adversarial Network. In ICLR.Google Scholar
38. Andreas Stolcke and Stephen Omohundro. 1994. Inducing Probabilistic Grammars by Bayesian Model Merging. In Proceedings of ICGI-94. 106–118. Google ScholarDigital Library
39. Jerry O. Talton, Yu Lou, Steve Lesser, Jared Duke, Radomír Měch, and Vladlen Koltun. 2011. Metropolis Procedural Modeling. ACM TOG 30, 2 (2011), 11:1–11:14. Google ScholarDigital Library
40. Jerry O. Talton, Lingfeng Yang, Ranjitha Kumar, Maxine Lim, Noah D. Goodman, and Radomír Měch. 2012. Learning Design Patterns with Bayesian Grammar Induction. In Proceedings of UIST ’12. 63–74. Google ScholarDigital Library
41. O. Teboul, I. Kokkinos, L. Simon, P. Koutsourakis, and N. Paragios. 2013. Parsing Facades with Shape Grammars and Reinforcement Learning. IEEE PAMI 35, 7 (July 2013). Google ScholarDigital Library
42. Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. 2015. Learning spatiotemporal features with 3d convolutional networks. In ICCV. Google ScholarDigital Library
43. Luan Tran, Xi Yin, and Xiaoming Liu. 2017. Disentangled Representation Learning GAN for Pose-Invariant Face Recognition. In CVPR.Google Scholar
44. Radim Tyleček and Radim Šára. 2013. Spatial Pattern Templates for Recognition of Objects with Regular Structure. In Proc. GCPR. Saarbrucken, Germany.Google ScholarCross Ref
45. Carlos A. Vanegas, Ignacio Garcia-Dorado, Daniel G. Aliaga, Bedrich Benes, and Paul Waddell. 2012. Inverse Design of Urban Procedural Models. ACM TOG 31, 6 (2012), 168:1–168:11. Google ScholarDigital Library
46. V. S. R. Veeravasarapu, Constantin A. Rothkopf, and Visvanathan Ramesh. 2017. Adversarially Tuned Scene Generation. (2017).Google Scholar
47. Chengde Wan, Thomas Probst, Luc Van Gool, and Angela Yao. 2017. Crossing Nets: Combining GANs and VAEs With a Shared Latent Space for Hand Pose Estimation. In CVPR.Google Scholar
48. David Warde-Farley and Yoshua Bengio. 2017. Improving Generative Adverarial Networks with Denoising Feature Matching. In CVPR.Google Scholar
49. Peter Wonka, Michael Wimmer, François Sillion, and William Ribarsky. 2003. Instant Architecture. ACM TOG 22, 3 (2003), 669–677. Google ScholarDigital Library
50. Raymond Yeh, Chen Chen, Teck-Yian Lim, Mark Hasegawa-Johnson, and Minh N. Do. 2016. Semantic Image Inpainting with Perceptual and Contextual Losses. (2016). arXiv:1607.07539Google Scholar
51. Yi-Ting Yeh, Katherine Breeden, Lingfeng Yang, Matthew Fisher, and Pat Hanrahan. 2013. Synthesis of Tiled Patterns using Factor Graphs. ACM TOG 32, 1 (2013). Google ScholarDigital Library
52. Junbo Zhao, Michael Mathieu, and Yann LeCun. 2017. Energy-based generative adversarial network. In ICLR.Google Scholar
53. Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A. Efros. 2016. Generative Visual Manipulation on the Natural Image Manifold. In ECCV.Google Scholar
54. Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. 2017a. Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks. ICCV (2017).Google Scholar
55. Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A Efros, Oliver Wang, and Eli Shechtman. 2017b. Toward Multimodal Image-to-Image Translation. In NIPS. Google ScholarDigital Library
