“Hierarchical Layout Blending with Recursive Optimal Correspondence” by Xu, Li, Yang, Shi, Fu, et al. …
Conference:
Type(s):
Title:
- Hierarchical Layout Blending with Recursive Optimal Correspondence
Session/Category Title: Geometric Operations
Presenter(s)/Author(s):
Abstract:
We present a novel method for blending hierarchical layouts with semantic labels. The core of our method is a hierarchical structure correspondence algorithm, which recursively finds optimal substructure correspondences, achieving a globally optimal correspondence between a pair of hierarchical layouts. This correspondence is consistent with the structures of both layouts, allowing us to define the union of the layouts’ structures. The resulting compound structure helps extract intermediate layout structures, from which blended layouts can be generated via an optimization approach. The correspondence also defines a similarity measure between layouts in a hierarchically structured view. Our method provides a new way for novel layout creation. The introduced structural similarity measure regularizes the layouts in a hyperspace. We demonstrate two applications in this paper, i.e., exploratory design of novel layouts and sketch-based layout retrieval, and test them on a magazine layout dataset. The effectiveness and feasibility of these two applications are confirmed by the user feedback and the extensive results. The code is available at https://github.com/lyf7115/LayoutBlending.
References:
1. Ibraheem Alhashim, Honghua Li, Kai Xu, Junjie Cao, Rui Ma, and Hao Zhang. 2014. Topology-varying 3D shape creation via structural blending. ACM Trans. Graph. 33, 4 (2014), 1–10.
2. Ibraheem Alhashim, Kai Xu, Yixin Zhuang, Junjie Cao, Patricio Simari, and Hao Zhang. 2015. Deformation-driven topology-varying 3D shape correspondence. ACM Trans. Graph. 34, 6 (2015), 1–13.
3. Rahul Arora, Ishan Darolia, Vinay P Namboodiri, Karan Singh, and Adrien Bousseau. 2017. Sketchsoup: Exploratory ideation using design sketches. Comput. Graph. Forum 36, 8 (2017), 302–312.
4. Diego Martin Arroyo, Janis Postels, and Federico Tombari. 2021. Variational Transformer Networks for Layout Generation. In CVPR. 13642–13652.
5. Greg J Badros, Alan Borning, and Peter J Stuckey. 2001. The Cassowary linear arithmetic constraint solving algorithm. ACM Trans. Comput.-Hum. Interact. 8, 4 (2001), 267–306.
6. Fan Bao, Dong-Ming Yan, Niloy J Mitra, and Peter Wonka. 2013. Generating and exploring good building layouts. ACM Trans. Graph. 32, 4 (2013), 1–10.
7. C Bradford Barber, David P Dobkin, and Hannu Huhdanpaa. 1996. The quickhull algorithm for convex hulls. ACM Trans. Math. Softw. 22, 4 (1996), 469–483.
8. Patrick Baudisch, Edward Cutrell, Ken Hinckley, and Adam Eversole. 2005. Snap-and-go: helping users align objects without the modality of traditional snapping. In CHI. 301–310.
9. Eric A Bier and Maureen C Stone. 1986. Snap-dragging. SIGGRAPH Comput. Graph. 20, 4 (1986), 233–240.
10. Alan Borning, Kim Marriott, Peter Stuckey, and Yi Xiao. 1997. Solving linear arithmetic constraints for user interface applications. In UIST. 87–96.
11. Sara Bunian, Kai Li, Chaima Jemmali, Casper Harteveld, Yun Fu, and Magy Seif Seif El-Nasr. 2021. VINS: Visual Search for Mobile User Interface Design. In CHI. 1–14.
12. Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. 2017. Rico: A mobile app dataset for building data-driven design applications. In UIST. 845–854.
13. Michael Goesele, Jens Ackermann, Simon Fuhrmann, Carsten Haubold, Ronny Klowsky, Drew Steedly, and Richard Szeliski. 2010. Ambient point clouds for view interpolation. ACM Trans. Graph. 29, 4 (2010), 1–6.
14. Gaël Guennebaud, Benoît Jacob, et al. 2010. Eigen v3. http://eigen.tuxfamily.org.
15. Kamal Gupta, Justin Lazarow, Alessandro Achille, Larry S Davis, Vijay Mahadevan, and Abhinav Shrivastava. 2021. Layouttransformer: Layout generation and completion with self-attention. In ICCV. 1004–1014.
16. Ruizhen Hu, Zeyu Huang, Yuhan Tang, Oliver Van Kaick, Hao Zhang, and Hui Huang. 2020. Graph2plan: Learning floorplan generation from layout graphs. ACM Trans. Graph. 39, 4 (2020), 118–1.
17. Forrest Huang, John F Canny, and Jeffrey Nichols. 2019. Swire: Sketch-based user interface retrieval. In CHI. 1–10.
18. Qi-Xing Huang, Bart Adams, Martin Wicke, and Leonidas J Guibas. 2008. Non-rigid registration under isometric deformations. Comput. Graph. Forum 27, 5 (2008), 1449–1457.
19. Arjun Jain, Thorsten Thormählen, Tobias Ritschel, and Hans-Peter Seidel. 2012. Exploring shape variations by 3d-model decomposition and part-based recombination. Comput. Graph. Forum 31, 2pt3 (2012), 631–640.
20. Yue Jiang, Ruofei Du, Christof Lutteroth, and Wolfgang Stuerzlinger. 2019. ORC layout: Adaptive GUI layout with OR-constraints. In CHI. 1–12.
21. Yue Jiang, Wolfgang Stuerzlinger, and Christof Lutteroth. 2021. ReverseORC: Reverse Engineering of Resizable User Interface Layouts with OR-Constraints. In CHI. 1–18.
22. Yue Jiang, Wolfgang Stuerzlinger, Matthias Zwicker, and Christof Lutteroth. 2020. ORCSolver: An Efficient Solver for Adaptive GUI Layout with OR-Constraints. In CHI. 1–14.
23. Akash Abdu Jyothi, Thibaut Durand, Jiawei He, Leonid Sigal, and Greg Mori. 2019. Layoutvae: Stochastic scene layout generation from a label set. In ICCV. 9895–9904.
24. Evangelos Kalogerakis, Siddhartha Chaudhuri, Daphne Koller, and Vladlen Koltun. 2012. A probabilistic model for component-based shape synthesis. ACM Trans. Graph. 31, 4 (2012), 1–11.
25. Youwen Kang, Zhida Sun, Sitong Wang, Zeyu Huang, Ziming Wu, and Xiaojuan Ma. 2021. MetaMap: Supporting Visual Metaphor Ideation through Multi-dimensional Example-based Exploration. In CHI. 1–15.
26. Kotaro Kikuchi, Edgar Simo-Serra, Mayu Otani, and Kota Yamaguchi. 2021. Constrained graphic layout generation via latent optimization. In ACM International Conference on Multimedia. 88–96.
27. Yanir Kleiman, Oliver van Kaick, Olga Sorkine-Hornung, and Daniel Cohen-Or. 2015. SHED: shape edit distance for fine-grained shape similarity. ACM Trans. Graph. 34, 6 (2015), 1–11.
28. Janin Koch and Antti Oulasvirta. 2016. Computational layout perception using gestalt laws. In CHI Extended Abstracts. 1423–1429.
29. Harold W Kuhn. 1955. The Hungarian method for the assignment problem. Naval Research Logistics Quarterly 2, 1–2 (1955), 83–97.
30. Ranjitha Kumar, Jerry O Talton, Salman Ahmad, and Scott R Klemmer. 2011. Bricolage: example-based retargeting for web design. In CHI. 2197–2206.
31. Hsin-Ying Lee, Lu Jiang, Irfan Essa, Phuong B Le, Haifeng Gong, Ming-Hsuan Yang, and Weilong Yang. 2020. Neural design network: Graphic layout generation with constraints. In ECCV. Springer, 491–506.
32. Jianan Li, Jimei Yang, Aaron Hertzmann, Jianming Zhang, and Tingfa Xu. 2019b. Layout-GAN: Generating Graphic Layouts with Wireframe Discriminators. In International Conference on Learning Representations.
33. Manyi Li, Akshay Gadi Patil, Kai Xu, Siddhartha Chaudhuri, Owais Khan, Ariel Shamir, Changhe Tu, Baoquan Chen, Daniel Cohen-Or, and Hao Zhang. 2019a. Grains: Generative recursive autoencoders for indoor scenes. ACM Trans. Graph. 38, 2 (2019), 1–16.
34. Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. 2015. SMPL: A skinned multi-person linear model. ACM Trans. Graph. 34, 6 (2015), 1–16.
35. Dipu Manandhar, Dan Ruta, and John Collomosse. 2020. Learning structural similarity of user interface layouts using graph networks. In ECCV. Springer, 730–746.
36. Akshay Gadi Patil, Omri Ben-Eliezer, Or Perel, and Hadar Averbuch-Elor. 2020. Read: Recursive autoencoders for document layout generation. In CVPR. 544–545.
37. Akshay Gadi Patil, Manyi Li, Matthew Fisher, Manolis Savva, and Hao Zhang. 2021. LayoutGMN: Neural Graph Matching for Structural Layout Similarity. In CVPR. 11048–11057.
38. Roope Raisamo and Kari-Jouko Räihä. 1996. A new direct manipulation technique for aligning objects in drawing programs. In UIST. 157–164.
39. William T Reeves. 1981. Inbetweening for computer animation utilizing moving point constraints. SIGGRAPH Comput. Graph. 15, 3 (1981), 263–269.
40. Alec Rivers, Takeo Igarashi, and Frédo Durand. 2010. 2.5 D cartoon models. ACM Trans. Graph. 29, 4 (2010), 1–7.
41. Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. 2008. The graph neural network model. IEEE Transactions on Neural Networks 20, 1 (2008), 61–80.
42. Stefan Schwarz, Mateusz Pawlik, and Nikolaus Augsten. 2017. A new perspective on the tree edit distance. In International Conference on Similarity Search and Applications. Springer, 156–170.
43. Wolfgang Stuerzlinger, Olivier Chapuis, Dusty Phillips, and Nicolas Roussel. 2006. User interface façades: towards fully adaptable user interfaces. In UIST. 309–318.
44. Sou Tabata, Hiroki Yoshihara, Haruka Maeda, and Kei Yokoyama. 2019. Automatic layout generation for graphical design magazines. In ACM SIGGRAPH 2019 Posters. 1–2.
45. Daniel Vlasic, Matthew Brand, Hanspeter Pfister, and Jovan Popović. 2005. Face Transfer with Multilinear Models. ACM Trans. Graph. 24, 3 (jul 2005), 426–433.
46. Josh Wills, Sameer Agarwal, David Kriegman, and Serge Belongie. 2009. Toward a perceptual space for gloss. ACM Trans. Graph. 28, 4 (2009), 1–15.
47. Wenming Wu, Xiao-Ming Fu, Rui Tang, Yuhan Wang, Yu-Hao Qi, and Ligang Liu. 2019. Data-driven interior plan generation for residential buildings. ACM Trans. Graph. 38, 6 (2019), 1–12.
48. Pengfei Xu, Hongbo Fu, Chiew-Lan Tai, and Takeo Igarashi. 2015. Gaca: Group-aware command-based arrangement of graphic elements. In CHI. 2787–2795.
49. Pengfei Xu, Guohang Yan, Hongbo Fu, Takeo Igarashi, Chiew-Lan Tai, and Hui Huang. 2019. Global Beautification of 2D and 3D Layouts with Interactive Ambiguity Resolution. IEEE TVCG (2019).
50. Kaizhong Zhang and Dennis Shasha. 1989. Simple fast algorithms for the editing distance between trees and related problems. SIAM J. Comput. 18, 6 (1989), 1245–1262.
51. Xinru Zheng, Xiaotian Qiao, Ying Cao, and Rynson WH Lau. 2019. Content-aware generative modeling of graphic design layouts. ACM Trans. Graph. 38, 4 (2019), 1–15.
52. Xu Zhong, Jianbin Tang, and Antonio Jimeno Yepes. 2019. Publaynet: largest dataset ever for document layout analysis. In International Conference on Document Analysis and Recognition. IEEE, 1015–1022.
53. Chenyang Zhu, Renjiao Yi, Wallace Lira, Ibraheem Alhashim, Kai Xu, and Hao Zhang. 2017. Deformation-driven shape correspondence via shape recognition. ACM Trans. Graph. 36, 4 (2017), 1–12.
