“TileGAN: synthesis of large-scale non-homogeneous textures” by Frühstück, Alhashim and Wonka
Conference:
Type(s):
Title:
- TileGAN: synthesis of large-scale non-homogeneous textures
Session/Category Title: High Performance Rendering
Presenter(s)/Author(s):
Abstract:
We tackle the problem of texture synthesis in the setting where many input images are given and a large-scale output is required. We build on recent generative adversarial networks and propose two extensions in this paper. First, we propose an algorithm to combine outputs of GANs trained on a smaller resolution to produce a large-scale plausible texture map with virtually no boundary artifacts. Second, we propose a user interface to enable artistic control. Our quantitative and qualitative results showcase the generation of synthesized high-resolution maps consisting of up to hundreds of megapixels as a case in point.
References:
1. Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Raffal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2015. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. (2015). tensorflow.org.Google Scholar
2. Adib Akl, Charles Yaacoub, Marc Donias, Jean-Pierre Da Costa, and Christian Germain. 2018. A survey of exemplar-based texture synthesis methods. Computer Vision and Image Understanding (April 2018).Google Scholar
3. Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. 2009. Patch-Match: A Randomized Correspondence Algorithm for Structural Image Editing. ACM Trans. Graph. 28, 3 (July 2009). Google ScholarDigital Library
4. Connelly Barnes, Fang-Lue Zhang, Liming Lou, Xian Wu, and Shi-Min Hu. 2015. PatchTable : Efficient Patch Queries for Large Datasets and Applications. ACM Trans. Graph. 34, 4 (July 2015). Google ScholarDigital Library
5. Heli Ben-Hamu, Haggai Maron, Itay Kezurer, Gal Avineri, and Yaron Lipman. 2018. Multi-chart Generative Surface Modeling. ACM Trans. Graph. 37, 6 (Dec. 2018). Google ScholarDigital Library
6. Urs Bergmann, Nikolay Jetchev, and Roland Vollgraf. 2017. Learning Texture Manifolds with the Periodic Spatial GAN. In Proceedings of the 34th International Conference on Machine Learning, Vol. 70. 469–477. Google ScholarDigital Library
7. Andrew Brock, Jeff Donahue, and Karen Simonyan. 2018. Large Scale GAN Training for High Fidelity Natural Image Synthesis. CoRR abs/1809.11096 (2018).Google Scholar
8. Kaidi Cao, Jing Liao, and Lu Yuan. 2018. CariGANs: Unpaired Photo-to-caricature Translation. ACM Trans. Graph. 37, 6 (Dec. 2018). Google ScholarDigital Library
9. Soheil Darabi, Eli Shechtman, Connelly Barnes, Dan B. Goldman, and Pradeep Sen. 2012. Image Melding: Combining Inconsistent Images Using Patch-based Synthesis. ACM Trans. Graph. 31, 4 (July 2012). Google ScholarDigital Library
10. Jeremy S. De Bonet. 1997. Multiresolution Sampling Procedure for Analysis and Synthesis of Texture Images. In Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’97). Google ScholarDigital Library
11. Alexey Dosovitskiy and Thomas Brox. 2016. Generating Images with Perceptual Similarity Metrics based on Deep Networks. In Advances in Neural Information Processing Systems 29. 658–666. Google ScholarDigital Library
12. Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. 2015a. A Neural Algorithm of Artistic Style. CoRR abs/1508.06576 (2015).Google Scholar
13. Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. 2015b. Texture Synthesis Using Convolutional Neural Networks. In Advances in Neural Information Processing Systems 28. Google ScholarDigital Library
14. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative Adversarial Nets. In Advances in Neural Information Processing Systems 27. Google ScholarDigital Library
15. Éric Guérin, Julie Digne, Éric Galin, Adrien Peytavie, Christian Wolf, Bedrich Benes, and Benoît Martinez. 2017. Interactive Example-based Terrain Authoring with Conditional Generative Adversarial Networks. ACM Trans. Graph. 36, 6 (Nov. 2017). Google ScholarDigital Library
16. Charles Han, Eric Risser, Ravi Ramamoorthi, and Eitan Grinspun. 2008. Multiscale Texture Synthesis. ACM Trans. Graph. 27, 3 (Aug. 2008). Google ScholarDigital Library
17. David J. Heeger and James R. Bergen. 1995. Pyramid-based Texture Analysis/Synthesis. In Proceedings of the 22Nd Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’95). Google ScholarDigital Library
18. Aaron Hertzmann, Charles E. Jacobs, Nuria Oliver, Brian Curless, and David H. Salesin. 2001. Image Analogies. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’01). Google ScholarDigital Library
19. Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. 2015. Single image super-resolution from transformed self-exemplars. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google ScholarCross Ref
20. Xun Huang, Ming-Yu Liu, Serge Belongie, and Jan Kautz. 2018. Multimodal Unsupervised Image-to-image Translation. In The European Conference on Computer Vision (ECCV).Google Scholar
21. Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017. Image-to-image translation with conditional adversarial networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google ScholarCross Ref
22. Ondřej Jamriška, Jakub Fišer, Paul Asente, Jingwan Lu, Eli Shechtman, and Daniel Sýkora. 2015. LazyFluids: Appearance Transfer for Fluid Animations. ACM Transactions on Graphics 34, 4, Article 92 (2015). Google ScholarDigital Library
23. Nikolay Jetchev, Urs Bergmann, and Calvin Seward. 2017. GANosaic: Mosaic Creation with Generative Texture Manifolds. CoRR abs/1712.00269 (2017).Google Scholar
24. Nikolay Jetchev, Urs Bergmann, and Roland Vollgraf. 2016. Texture synthesis with spatial generative adversarial networks. CoRR abs/1611.08207 (2016).Google Scholar
25. Nikolay Jetchev, Urs Bergmann, and Gokhan Yildirim. 2018. Copy the Old or Paint Anew? An Adversarial Framework for (non-) Parametric Image Stylization. CoRR abs/1811.09236 (2018).Google Scholar
26. Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. 2018a. Progressive Growing of GANs for Improved Quality, Stability, and Variation. In International Conference on Learning Representations.Google Scholar
27. Tero Karras, Samuli Laine, and Timo Aila. 2018b. A Style-Based Generator Architecture for Generative Adversarial Networks. CoRR abs/1811.09236 (2018).Google Scholar
28. Alexandre Kaspar, Boris Neubert, Dani Lischinski, Mark Pauly, and Johannes Kopf. 2015. Self Tuning Texture Optimization. Computer Graphics Forum 34, 2 (2015). Google ScholarDigital Library
29. Tom Kelly, Paul Guerrero, Anthony Steed, Peter Wonka, and Niloy J. Mitra. 2018. FrankenGAN: Guided Detail Synthesis for Building Mass Models Using Style-Synchonized GANs. ACM Trans. Graph. 37, 6 (2018). Google ScholarDigital Library
30. Vivek Kwatra, Irfan Essa, Aaron Bobick, and Nipun Kwatra. 2005. Texture Optimization for Example-based Synthesis. ACM Trans. Graph. 24, 3 (July 2005). Google ScholarDigital Library
31. Vivek Kwatra, Arno Schödl, Irfan Essa, Greg Turk, and Aaron Bobick. 2003. Graphcut Textures: Image and Video Synthesis Using Graph Cuts. ACM Trans. Graph. 22, 3 (July 2003). Google ScholarDigital Library
32. Sylvain Lefebvre and Hugues Hoppe. 2005. Parallel Controllable Texture Synthesis. ACM Trans. Graph. 24, 3 (July 2005). Google ScholarDigital Library
33. Chuan Li and Michael Wand. 2016. Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
34. Yitzchak David Lockerman, Basile Sauvage, Rémi Allègre, Jean-Michel Dischler, Julie Dorsey, and Holly Rushmeier. 2016. Multi-scale Label-map Extraction for Texture Synthesis. ACM Trans. Graph. 35, 4 (July 2016). Google ScholarDigital Library
35. Michal Lukáč, Jakub Fišer, Paul Asente, Jingwan Lu, Eli Shechtman, and Daniel Sýkora. 2015. Brushables: Example-based Edge-aware Directional Texture Painting. Computer Graphics Forum 34, 7 (2015). Google ScholarDigital Library
36. Roey Mechrez, Itamar Talmi, and Lihi Zelnik-Manor. 2018. The Contextual Loss for Image Transformation with Non-aligned Data. In ECCV.Google Scholar
37. Koki Nagano, Jaewoo Seo, Jun Xing, Lingyu Wei, Zimo Li, Shunsuke Saito, Aviral Agarwal, Jens Fursund, and Hao Li. 2018. paGAN: Real-time Avatars Using Dynamic Textures. ACM Trans. Graph. 37, 6 (Dec. 2018). Google ScholarDigital Library
38. Deepak Pathak, Philipp Krähenbühl, Jeff Donahue, Trevor Darrell, and Alexei A. Efros. 2016. Context Encoders: Feature Learning by Inpainting. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
39. Javier Portilla and Eero P Simoncelli. 2000. A parametric texture model based on joint statistics of complex wavelet coefficients. International Journal of Computer Vision 40, 1 (2000). Google ScholarDigital Library
40. Lincoln Ritter, Wilmot Li, Brian Curless, Maneesh Agrawala, and David Salesin. 2006. Painting with Texture. In Proceedings of the 17th Eurographics Conference on Rendering Techniques (EGSR ’06). Google ScholarDigital Library
41. Omry Sendik and Daniel Cohen-Or. 2017. Deep Correlations for Texture Synthesis. ACM Trans. Graph. 36, 4 (July 2017).Google ScholarDigital Library
42. Xavier Snelgrove. 2017. High-resolution Multi-scale Neural Texture Synthesis. In SIGGRAPH Asia 2017 Technical Briefs. Google ScholarDigital Library
43. Hao Wang, Nadav Schor, Ruizhen Hu, Haibin Huang, Daniel Cohen-Or, and Hui Huang. 2018a. Global-to-local Generative Model for 3D Shapes. ACM Trans. Graph. 37, 6 (Dec. 2018). Google ScholarDigital Library
44. Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Chen Change Loy, Yu Qiao, and Xiaoou Tang. 2018c. ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks. ECCV workshops (2018).Google Scholar
45. Yi Wang, Xin Tao, Xiaojuan Qi, Xiaoyong Shen, and Jiaya Jia. 2018b. Image Inpainting via Generative Multi-column Convolutional Neural Networks. In Advances in Neural Information Processing Systems. 329–338. Google ScholarDigital Library
46. Li-Yi Wei, Sylvain Lefebvre, Vivek Kwatra, and Greg Turk. 2009. State of the Art in Example-based Texture Synthesis. In Eurographics 2009, State of the Art Report, EG-STAR. Eurographics Association.Google Scholar
47. Raymond A. Yeh, Chen Chen, Teck Yian Lim, Schwing Alexander G., Mark Hasegawa-Johnson, and Minh N. Do. 2017. Semantic Image Inpainting with Deep Generative Models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
48. Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. 2018. Generative Image Inpainting with Contextual Attention. CoRR abs/1801.07892 (2018).Google Scholar
49. Han Zhang, Ian J. Goodfellow, Dimitris N. Metaxas, and Augustus Odena. 2018. Self-Attention Generative Adversarial Networks. CoRR abs/1805.08318 (2018).Google Scholar
50. Yang Zhou, Huajie Shi, Dani Lischinski, Minglun Gong, Johannes Kopf, and Hui Huang. 2017. Analysis and Controlled Synthesis of Inhomogeneous Textures. Computer Graphics Forum 36, 2 (2017). Google ScholarDigital Library
51. Yang Zhou, Zhen Zhu, Xiang Bai, Dani Lischinski, Daniel Cohen-Or, and Hui Huang. 2018. Non-stationary Texture Synthesis by Adversarial Expansion. ACM Trans. Graph. 37, 4 (July 2018). Google ScholarDigital Library
52. Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. 2017a. Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks. In 2017 IEEE International Conference on Computer Vision (ICCV).Google Scholar
53. Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A Efros, Oliver Wang, and Eli Shechtman. 2017b. Toward Multimodal Image-to-Image Translation. In Advances in Neural Information Processing Systems 30. Google ScholarDigital Library
