“Content-aware generative modeling of graphic design layouts” by Zheng, Qiao, Cao and Lau
Conference:
Type:
Title:
- Content-aware generative modeling of graphic design layouts
Session/Category Title: Design and Layout
Presenter(s)/Author(s):
Abstract:
Layout is fundamental to graphic designs. For visual attractiveness and efficient communication of messages and ideas, graphic design layouts often have great variation, driven by the contents to be presented. In this paper, we study the problem of content-aware graphic design layout generation. We propose a deep generative model for graphic design layouts that is able to synthesize layout designs based on the visual and textual semantics of user inputs. Unlike previous approaches that are oblivious to the input contents and rely on heuristic criteria, our model captures the effect of visual and textual contents on layouts, and implicitly learns complex layout structure variations from data without the use of any heuristic rules. To train our model, we build a large-scale magazine layout dataset with fine-grained layout annotations and keyword labeling. Experimental results show that our model can synthesize high-quality layouts based on the visual semantics of input images and keyword-based summary of input text. We also demonstrate that our model internally learns powerful features that capture the subtle interaction between contents and layouts, which are useful for layout-aware design retrieval.
References:
1. Apostolos Antonacopoulos, David Bridson, Christos Papadopoulos, and Stefan Pletschacher. 2009. A realistic dataset for performance evaluation of document layout analysis. In Proc. ICDAR. 296–300. Google ScholarDigital Library
2. Michael W Berry and Jacob Kogan. 2010. Text Mining: Applications and Theory. John Wiley & Sons.Google ScholarCross Ref
3. Andrew Brock, Theodore Lim, JM Ritchie, and Nick Weston. 2017. Neural photo editing with introspective adversarial networks. In Proc. ICLR.Google Scholar
4. Zoya Bylinskii, Nam Wook Kim, Peter O’Donovan, Sami Alsheikh, Spandan Madan, Hanspeter Pfister, Fredo Durand, Bryan Russell, and Aaron Hertzmann. 2017. Learning visual importance for graphic designs and data visualizations. In Proc. ACM UIST. 57–69. Google ScholarDigital Library
5. Ying Cao, Antoni Chan, and Rynson Lau. 2012. Automatic stylistic manga layout. ACM TOG 31, 6 (2012). Google ScholarDigital Library
6. Ying Cao, Rynson Lau, and Antoni Chan. 2014. Look Over Here: Attention-Directing Composition of Manga Elements. ACM TOG 33, 4 (2014). Google ScholarDigital Library
7. Jaime Carbonell and Jade Goldstein. 1998. The use of MMR, diversity-based reranking for reordering documents and producing summaries. In Proc. ACM SIGIR. 335–336. Google ScholarDigital Library
8. Niranjan Damera-Venkata, Josép Bento, and Eamonn O’Brien-Strain. 2011. Probabilistic document model for automated document composition. In Proc. ACM DocEng. 3–12. Google ScholarDigital Library
9. Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. 2017. Adversarial feature learning. In Proc. ICLR.Google Scholar
10. Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Alex Lamb, Martin Arjovsky, Olivier Mastropietro, and Aaron Courville. 2017. Adversarially learned inference. In Proc. ICLR.Google Scholar
11. SM Ali Eslami, Nicolas Heess, Christopher Williams, and John Winn. 2014. The shape Boltzmann machine: A strong model of object shape. IJCV 107, 2 (2014), 155–176. Google ScholarDigital Library
12. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Proc. NIPS. 2672–2680. Google ScholarDigital Library
13. Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi. 2018. Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks. In Proc. IEEE CVPR.Google ScholarCross Ref
14. Bernardo Heynemann, Cezar Espinola, and Fabio M. Costa. 2015. Detection Algorithms. http://thumbor.readthedocs.io/en/latest/detection_algorithms.html.Google Scholar
15. Geoffrey Hinton, Simon Osindero, and YeeWhye Teh. 2006. A fast learning algorithm for deep belief nets. Neural Computation 18, 7 (2006), 1527–1554. Google ScholarDigital Library
16. Nathan Hurst, Wilmot Li, and Kim Marriott. 2009. Review of automatic document formatting. In Proc. ACM DocEng. 99–108. Google ScholarDigital Library
17. Phillip Isola, JunYan Zhu, Tinghui Zhou, and Alexei Efros. 2017. Image-to-image translation with conditional adversarial networks. In Proc. CVPR.Google ScholarCross Ref
18. Charles Jacobs, Wilmot Li, Evan Schrier, David Bargeron, and David Salesin. 2003. Adaptive grid-based document layout. ACM TOG 22, 3 (2003), 838–847. Google ScholarDigital Library
19. Tom Kelly, Paul Guerrero, Anthony Steed, Peter Wonka, and Niloy J Mitra. 2018. FrankenGAN: Guided Detail Synthesis for Building Mass-Models Using Style-Synchonized GANs. ACM TOG (2018). Google ScholarDigital Library
20. Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proc. ICLR.Google Scholar
21. Diederik Kingma and Max Welling. 2014. Auto-encoding variational bayes. In Proc. ICLR.Google Scholar
22. Ranjitha Kumar, Jerry Talton, Salman Ahmad, and Scott Klemmer. 2011. Bricolage: example-based retargeting for Web design. In Proc. ACM CHI. 2197–2206. Google ScholarDigital Library
23. Jun Li, Kai Xu, Siddhartha Chaudhuri, Ersin Yumer, Hao Zhang, and Leonidas Guibas. 2017. GRASS: Generative Recursive Autoencoders for Shape Structures. ACM TOG 36, 4 (2017). Google ScholarDigital Library
24. Jianan Li, Tingfa Xu, Jianming Zhang, Aaron Hertzmann, and Jimei Yang. 2019. LayoutGAN: Generating Graphic Layouts with Wireframe Discriminator. In Proc. ICLR.Google Scholar
25. Jonathan Long, Evan Shelhamer, and Trevor Darrell. 2015. Fully convolutional networks for semantic segmentation. In Proc. CVPR. 3431–3440.Google ScholarCross Ref
26. Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. 2017. Least squares generative adversarial networks. In Proc. ICCV. 2813–2821.Google ScholarCross Ref
27. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proc. NIPS. 3111–3119. Google ScholarDigital Library
28. Peter O’Donovan, Aseem Agarwala, and Aaron Hertzmann. 2014. Learning Layouts for Single-Page Graphic Designs. IEEE TVCG 20, 8 (2014), 1200–1213. Google ScholarDigital Library
29. Peter O’Donovan, Aseem Agarwala, and Aaron Hertzmann. 2015. DesignScape: Design with Interactive Layout Suggestions. In Proc. ACM CHI. 1221–1224. Google ScholarDigital Library
30. Xufang Pang, Ying Cao, Rynson Lau, and Antoni Chan. 2016. Directing user attention via visual flow on web designs. ACM TOG 35, 6 (2016). Google ScholarDigital Library
31. ZA Prust. 2010. Graphic Communications. Goodheart-Wilcox Publisher.Google Scholar
32. Alec Radford, Luke Metz, and Soumith Chintala. 2015. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv:1511.06434 (2015).Google Scholar
33. Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. 2014. Stochastic backpropagation and approximate inference in deep generative models. In Proc. ICML. Google ScholarDigital Library
34. Neha Saleem. 2015. The key characteristics of a fashion magazine. http://nehasaleemmedia.weebly.com/blog/the-key-characteristics-of-a-fashion-magazine.Google Scholar
35. Evan Schrier, Mira Dontcheva, Charles Jacobs, Geraldine Wade, and David Salesin. 2008. Adaptive layout for dynamically aggregated documents. In Proc. IUI. 99–108. Google ScholarDigital Library
36. Karen Simonyan and Andrew Zisserman. 2015. Very deep convolutional networks for large-scale image recognition. In Proc. ICLR.Google Scholar
37. David Smith. 2014. Magazine Design Tips: Key Elements. https://www.envision-creative.com/magazine-design-tips-key-elements/.Google Scholar
38. Adobe Spark. 2018. https://spark.adobe.com/.Google Scholar
39. Mary Stribley. 2015. 10 Rules of Composition All Designers Live By. https://designschool.canva.com/blog/visual-design-composition/.Google Scholar
40. Kashyap Todi, Daryl Weir, and Antti Oulasvirta. 2016. Sketchplore: Sketch and explore with a layout optimiser. In Proc. ACM DIS. 543–555. Google ScholarDigital Library
41. Leon Todoran, Marcel Worring, and Arnold WM Smeulders. 2005. The UvA color document dataset. International Journal on Document Analysis and Recognition 7, 4 (2005), 228–240. Google ScholarDigital Library
42. Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 2015. 3d shapenets: A deep representation for volumetric shapes. In Proc. IEEE CVPR.Google Scholar
43. Xuyong Yang, Tao Mei, Ying-Qing Xu, Yong Rui, and Shipeng Li. 2016. Automatic generation of visual-textual presentation layout. ACM TOMM 12, 2 (2016), 33. Google ScholarDigital Library
44. Charles Ying. 2014. Automating Layouts Bring Flipboard’s Magazine Style To Web And Windows. https://techcrunch.com/2014/03/23/layout-in-flipboard-for-web-and-windows/?ncid=rss.Google Scholar
45. Nanxuan Zhao, Ying Cao, and Rynson Lau. 2018. What Characterizes Personalities of Graphic Designs? ACM TOG 37, 4 (2018), 1–15. Google ScholarDigital Library
46. Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei Efros. 2016. Generative visual manipulation on the natural image manifold. In Proc. ECCV. 597–613.Google ScholarCross Ref
