“Intuitive and efficient roof modeling for reconstruction and synthesis” by Ren, Zhang, Wu, Huang, Fan, et al. …
Conference:
Type(s):
Title:
- Intuitive and efficient roof modeling for reconstruction and synthesis
Session/Category Title: Reconstruction
Presenter(s)/Author(s):
Abstract:
We propose a novel and flexible roof modeling approach that can be used for constructing planar 3D polygon roof meshes. Our method uses a graph structure to encode roof topology and enforces the roof validity by optimizing a simple but effective planarity metric we propose. This approach is significantly more efficient than using general purpose 3D modeling tools such as 3ds Max or SketchUp, and more powerful and expressive than specialized tools such as the straight skeleton. Our optimization-based formulation is also flexible and can accommodate different styles and user preferences for roof modeling. We showcase two applications. The first application is an interactive roof editing framework that can be used for roof design or roof reconstruction from aerial images. We highlight the efficiency and generality of our approach by constructing a mesh-image paired dataset consisting of 2539 roofs. Our second application is a generative model to synthesize new roof meshes from scratch. We use our novel dataset to combine machine learning and our roof optimization techniques, by using transformers and graph convolutional networks to model roof topology, and our roof optimization methods to enforce the planarity constraint.
References:
1. Oswin Aichholzer and Franz Aurenhammer. 1996. Straight skeletons for general polygonal figures in the plane. In International Computing and Combinatorics Conference. Springer, 117–126.
2. Oswin Aichholzer, Franz Aurenhammer, David Alberts, and Bernd Gärtner. 1996. A novel type of skeleton for polygons. (1996), 752–761.
3. F Alidoost, H Arefi, and M Hahn. 2020. Y-Shaped convolutional neural network for 3D roof elements extraction to reconstruct building models from a single aerial image. ISPRS Annals of Photogrammetry, Remote Sensing & Spatial Information Sciences 5, 2 (2020).
4. Murat Arikan, Michael Schwärzler, Simon Flöry, Michael Wimmer, and Stefan Maierhofer. 2013. O-snap: Optimization-based snapping for modeling architecture. ACM Transactions on Graphics (TOG) 32, 1 (2013), 1–15.
5. Jean-Philippe Bauchet and Florent Lafarge. 2020. Kinetic shape reconstruction. ACM Transactions on Graphics (TOG) 39, 5 (2020), 1–14.
6. Therese Biedl, Martin Held, Stefan Huber, Dominik Kaaser, and Peter Palfrader. 2015. Weighted straight skeletons in the plane. Computational Geometry 48, 2 (2015), 120–133.
7. Andrew Brock, Theodore Lim, James Millar Ritchie, and Nicholas J Weston. 2016. Generative and Discriminative Voxel Modeling with Convolutional Neural Networks. In Neural Inofrmation Processing Conference: 3D Deep Learning.
8. Cyprien Buron, Jean-Eudes Marvie, and Pascal Gautron. 2013. GPU Roof Grammars. In Eurographics (Short Papers). 85–88.
9. Xi Chen, Nikhil Mishra, Mostafa Rohaninejad, and Pieter Abbeel. 2018. Pixelsnail: An improved autoregressive generative model. In International Conference on Machine Learning. PMLR, 864–872.
10. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc Viet Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive Language Models beyond a Fixed-Length Context. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, Anna Korhonen, David R. Traum, and Lluís Màrquez (Eds.). Association for Computational Linguistics, 2978–2988.
11. Youness Dehbi, André Henn, Gerhard Gröger, Viktor Stroh, and Lutz Plümer. 2021. Robust and fast reconstruction of complex roofs with active sampling from 3D point clouds. Transactions in GIS 25, 1 (2021), 112–133.
12. Ilke Demir, Daniel G Aliaga, and Bedrich Benes. 2015. Procedural editing of 3d building point clouds. In Proceedings of the IEEE International Conference on Computer Vision (ICCV). 2147–2155.
13. Laurent Dinh, David Krueger, and Yoshua Bengio. 2015. NICE: Non-linear Independent Components Estimation. In International Conference on Learning Representations (ICLR), Yoshua Bengio and Yann LeCun (Eds.).
14. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An Image is Worth 16×16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations (ICLR).
15. Günther Eder and Martin Held. 2018. Computing positively weighted straight skeletons of simple polygons based on a bisector arrangement. Inform. Process. Lett. 132 (2018), 28–32.
16. David Eppstein and Jeff Erickson. 1999. Raising roofs, crashing cycles, and playing pool: Applications of a data structure for finding pairwise interactions. Discrete & Computational Geometry 22, 4 (1999), 569–592.
17. Petr Felkel and Stepan Obdrzalek. 1998. Straight skeleton implementation. In Proceedings of Spring Conference on Computer Graphics. Citeseer.
18. Matthew Fisher, Manolis Savva, Yangyan Li, Pat Hanrahan, and Matthias Nießner. 2015. Activity-centric Scene Synthesis for Functional 3D Scene Modeling. ACM Transactions on Graphics (TOG) 34, 6 (2015).
19. Lin Gao, Jie Yang, Tong Wu, Yu-Jie Yuan, Hongbo Fu, Yu-Kun Lai, and Hao(Richard) Zhang. 2019. SDM-NET: Deep Generative Network for Structured Deformable Mesh. ACM Transactions on Graphics (TOG) 38, 6 (2019), 243:1–243:15.
20. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. Advances in Neural Information Processing Systems 27 (2014), 2672–2680.
21. Martin Habbecke and Leif Kobbelt. 2012. Linear analysis of nonlinear constraints for interactive geometric modeling. In Computer Graphics Forum, Vol. 31. Wiley Online Library, 641–650.
22. Martin Held and Peter Palfrader. 2017. Straight skeletons with additive and multiplicative weights and their application to the algorithmic generation of roofs and terrains. Computer-Aided Design 92 (2017), 33–41.
23. Ruizhen Hu, Zeyu Huang, Yuhan Tang, Oliver van Kaick, Hao Zhang, and Hui Huang. 2020. Graph2Plan: Learning Floorplan Generation from Layout Graphs. arXiv preprint arXiv:2004.13204 (2020).
24. Caigui Jiang, Chengcheng Tang, Amir Vaxman, Peter Wonka, and Helmut Pottmann. 2015. Polyhedral Patterns. ACM Transactions On Graphics (TOG) 34, 6, Article 172 (Oct. 2015), 12 pages.
25. Tom Kelly, John Femiani, Peter Wonka, and Niloy J Mitra. 2017. BigSUR: large-scale structured urban reconstruction. ACM Transactions On Graphics (TOG) 36, 6 (2017).
26. Tom Kelly, Paul Guerrero, Anthony Steed, Peter Wonka, and Niloy J Mitra. 2018. FrankenGAN: Guided detail synthesis for building mass models using style-Synchonized Gans. ACM Transactions On Graphics (TOG) 37, 6 (2018), 1–14.
27. Tom Kelly and Peter Wonka. 2011. Interactive architectural modeling with procedural extrusions. ACM Transactions on Graphics (TOG) 30, 2 (2011), 1–15.
28. Hyeongju Kim, Hyeonseung Lee, Woo Hyun Kang, Joun Yeop Lee, and Nam Soo Kim. 2020. SoftFlow: Probabilistic Framework for Normalizing Flow on Manifolds. Advances in Neural Information Processing Systems 33 (2020).
29. Diederik P. Kingma and Max Welling. 2014. Auto-Encoding Variational Bayes. In International Conference on Learning Representations (ICLR), Yoshua Bengio and Yann LeCun (Eds.).
30. Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 (2016).
31. Mathieu Larive and Veronique Gaildrat. 2006. Wall grammar for building generation. In Proceedings of the 4th international conference on Computer graphics and interactive techniques in Australasia and Southeast Asia. 429–437.
32. Robert G Laycock and AM Day. 2003. Automatically generating roof models from building footprints. (2003).
33. Hui Lin, Jizhou Gao, Yu Zhou, Guiliang Lu, Mao Ye, Chenxi Zhang, Ligang Liu, and Ruigang Yang. 2013. Semantic decomposition and reconstruction of residential scenes from LiDAR data. ACM Transactions on Graphics (TOG) 32, 4 (2013), 1–10.
34. Chen Liu, Jimei Yang, Duygu Ceylan, Ersin Yumer, and Yasutaka Furukawa. 2018. Planenet: Piece-wise planar reconstruction from a single rgb image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2579–2588.
35. Yang Liu, Helmut Pottmann, Johannes Wallner, Yong-Liang Yang, and Wenping Wang. 2006. Geometric modeling with conical meshes and developable surfaces. In ACM Transactions On Graphics (TOG). 681–689.
36. Paul Merrell, Eric Schkufza, and Vladlen Koltun. 2010. Computer-generated residential building layouts. In ACM Transactions On Graphics (TOG). 1–12.
37. Kaichun Mo, Paul Guerrero, Li Yi, Hao Su, Peter Wonka, Niloy Mitra, and Leonidas Guibas. 2019. StructureNet: Hierarchical Graph Networks for 3D Shape Generation. ACM Transactions on Graphics (TOG) 38, 6 (2019), Article 242.
38. Pascal Müller, Peter Wonka, Simon Haegler, Andreas Ulmer, and Luc Van Gool. 2006. Procedural modeling of buildings. In ACM Transactions On Graphics (TOG). 614–623.
39. Przemyslaw Musialski, Peter Wonka, Daniel G Aliaga, Michael Wimmer, Luc Van Gool, and Werner Purgathofer. 2013. A survey of urban reconstruction. In Computer Graphics Forum, Vol. 32. Wiley Online Library, 146–177.
40. Liangliang Nan and Peter Wonka. 2017. Polyfit: Polygonal surface reconstruction from point clouds. In Proceedings of the IEEE International Conference on Computer Vision (ICCV). 2353–2361.
41. Charlie Nash, Yaroslav Ganin, S. M. Ali Eslami, and Peter W. Battaglia. 2020. PolyGen: An Autoregressive Generative Model of 3D Meshes. In Proceedings of the 37th International Conference on Machine Learning (ICML) (Proceedings of Machine Learning Research), Vol. 119. PMLR, 7220–7229.
42. Wamiq Reyaz Para, Paul Guerrero, Tom Kelly, Leonidas J. Guibas, and Peter Wonka. 2020. Generative Layout Modeling using Constraint Graphs. CoRR abs/2011.13417 (2020).
43. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. Fox, and R. Garnett (Eds.). Curran Associates, Inc., 8024–8035.
44. Helmut Pottmann, Yang Liu, Johannes Wallner, Alexander Bobenko, and Wenping Wang. 2007. Geometry of multi-layer freeform structures for architecture. In ACM Transactions On Graphics (TOG). 65–es.
45. Helmut Pottmann, Alexander Schiftner, Pengbo Bo, Heinz Schmiedhofer, Wenping Wang, Niccolo Baldassini, and Johannes Wallner. 2008. Freeform surfaces from single curved panels. ACM Transactions on Graphics (TOG) 27, 3 (2008), 1–10.
46. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. (2019).
47. Anurag Ranjan, Timo Bolkart, Soubhik Sanyal, and Michael J Black. 2018. Generating 3D faces using convolutional mesh autoencoders. In Proceedings of the European Conference on Computer Vision (ECCV). 704–720.
48. Ali Razavi, Aaron van den Oord, and Oriol Vinyals. 2019. Generating diverse high-fidelity images with vq-vae-2. In Advances in Neural Information Processing Systems. 14866–14876.
49. Jing Ren, Jens Schneider, Maks Ovsjanikov, and Peter Wonka. 2018. Joint Graph Layouts for Visualizing Collections of Segmented Meshes. IEEE Transactions on Visualization and Computer Graphics 24, 9 (2018), 2546–2558.
50. Danilo Rezende and Shakir Mohamed. 2015. Variational Inference with Normalizing Flows. In International Conference on Machine Learning. 1530–1538.
51. Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P. Kingma. 2017. PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications. In International Conference on Learning Representations (ICLR). Open-Review.net.
52. David Salinas, Florent Lafarge, and Pierre Alliez. 2015. Structure-aware mesh decimation. In Computer Graphics Forum, Vol. 34. Wiley Online Library, 211–227.
53. Michał Stypułkowski, Maciej Zamorski, Maciej Zięba, and Jan Chorowski. 2019. Conditional invertible flow for point cloud generation. arXiv preprint arXiv:1910.07344 (2019).
54. Kenichi Sugihara. 2013. Straight skeleton for automatic generation of 3-D building models with general shaped roofs. (2013).
55. Kenichi Sugihara. 2019. Straight Skeleton Computation Optimized for Roof Model Generation. In WSCG, Vol. 27. 101–109.
56. Qingyang Tan, Lin Gao, Yu-Kun Lai, and Shihong Xia. 2018. Variational autoencoders for deforming 3d mesh models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 5841–5850.
57. Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. 2016. Conditional image generation with pixelcnn decoders. Advances in Neural Information Processing Systems 29 (2016), 4790–4798.
58. Aaron Van Den Oord, Oriol Vinyals, et al. 2017. Neural discrete representation learning. In Advances in Neural Information Processing Systems. 6306–6315.
59. Aaron Van Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. 2016. Pixel Recurrent Neural Networks. In International Conference on Machine Learning. 1747–1756.
60. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems. 5998–6008.
61. Yannick Verdie, Florent Lafarge, and Pierre Alliez. 2015. LOD generation for urban scenes. ACM Transactions On Graphics (TOG) 34, ARTICLE (2015), 30.
62. Xinpeng Wang, Chandan Yeshwanth, and Matthias NieSSner. 2020. SceneFormer: Indoor Scene Generation with Transformers. arXiv:cs.CV/2012.09793
63. Jiajun Wu, Chengkai Zhang, Tianfan Xue, William T Freeman, and Joshua B Tenenbaum. 2016. Learning a probabilistic latent space of object shapes via 3D generative-adversarial modeling. In Proceedings of the 30th International Conference on Neural Information Processing Systems. 82–90.
64. Guandao Yang, Xun Huang, Zekun Hao, Ming-Yu Liu, Serge Belongie, and Bharath Hariharan. 2019. Pointflow: 3d point cloud generation with continuous normalizing flows. In Proceedings of the IEEE International Conference on Computer Vision (ICCV). 4541–4550.
65. Jie Yang, Kaichun Mo, Yu-Kun Lai, Leonidas J. Guibas, and Lin Gao. 2020. DSM-Net: Disentangled Structured Mesh Net for Controllable Generation of Fine Geometry. arXiv:cs.GR/2008.05440
66. Dawen Yu, Shunping Ji, Jin Liu, and Shiqing Wei. 2021. Automatic 3D building reconstruction from multi-view aerial images with deep learning. ISPRS Journal of Photogrammetry and Remote Sensing 171 (2021), 155–170.
67. Lap Fai Yu, Sai Kit Yeung, Chi Keung Tang, Demetri Terzopoulos, Tony F Chan, and Stanley J Osher. 2011. Make it home: automatic optimization of furniture arrangement. ACM Transactions on Graphics (TOG) 30, 4 (2011).
68. Huayi Zeng, Jiaye Wu, and Yasutaka Furukawa. 2018. Neural procedural reconstruction for residential buildings. In Proceedings of the European Conference on Computer Vision (ECCV). 737–753.
69. Fuyang Zhang, Nelson Nauata, and Yasutaka Furukawa. 2020. Conv-mpn: Convolutional message passing neural network for structured outdoor architecture reconstruction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2798–2807.
70. Qian-Yi Zhou and Ulrich Neumann. 2008. Fast and extensible building modeling from airborne LiDAR data. In Proceedings of the 16th ACM SIGSPATIAL international conference on Advances in geographic information systems. 1–8.
71. Qian-Yi Zhou and Ulrich Neumann. 2010. 2.5 d dual contouring: A robust approach to creating building models from aerial lidar point clouds. In Proceedings of the European Conference on Computer Vision (ECCV). Springer, 115–128.
72. Qian-Yi Zhou and Ulrich Neumann. 2011. 2.5 D building modeling with topology control. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2489–2496.
73. Lingjie Zhu, Shuhan Shen, Xiang Gao, and Zhanyi Hu. 2018. Large scale urban scene modeling from MVS meshes. In Proceedings of the European Conference on Computer Vision (ECCV). 614–629.
