“SDM-NET: deep generative network for structured deformable mesh” by Gao, Yang, Wu, Yuan, Fu, et al. …
Conference:
Type(s):
Title:
- SDM-NET: deep generative network for structured deformable mesh
Session/Category Title: Geometry Off the Deep End
Presenter(s)/Author(s):
Moderator(s):
Abstract:
We introduce SDM-NET, a deep generative neural network which produces structured deformable meshes. Specifically, the network is trained to generate a spatial arrangement of closed, deformable mesh parts, which respects the global part structure of a shape collection, e.g., chairs, airplanes, etc. Our key observation is that while the overall structure of a 3D shape can be complex, the shape can usually be decomposed into a set of parts, each homeomorphic to a box, and the finer-scale geometry of the part can be recovered by deforming the box. The architecture of SDM-NET is that of a two-level variational autoencoder (VAE). At the part level, a PartVAE learns a deformable model of part geometries. At the structural level, we train a Structured Parts VAE (SP-VAE), which jointly learns the part structure of a shape collection and the part geometries, ensuring the coherence between global shape structure and surface details. Through extensive experiments and comparisons with the state-of-the-art deep generative models of shapes, we demonstrate the superiority of SDM-NET in generating meshes with visual quality, flexible topology, and meaningful structures, benefiting shape interpolation and other subsequent modeling tasks.
References:
1. Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. 2018. Learning Representations and Generative Models for 3D Point Clouds. In International Conference on Machine Learning (ICML), Vol. 80. 40–49.Google Scholar
2. Dragomir Anguelov, Praveen Srinivasan, Daphne Koller, Sebastian Thrun, Jim Rodgers, and James Davis. 2005. SCAPE: shape completion and animation of people. ACM Trans. Graph. 24, 3 (2005), 408–416.Google ScholarDigital Library
3. Melinos Averkiou, Vladimir G Kim, Youyi Zheng, and Niloy J Mitra. 2014. Shapesynth: Parameterizing model collections for coupled shape exploration and synthesis. Comp. Graph. Forum 33, 2 (2014), 125–134.Google ScholarDigital Library
4. Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. 2015. ShapeNet: An Information-Rich 3D Model Repository. Technical Report arXiv:1512.03012.Google Scholar
5. Siddhartha Chaudhuri, Daniel Ritchie, Kai Xu, and Hao Zhang. 2019. Learning Generative Models of 3D Structures. In Eurographics Tutorial.Google Scholar
6. Zhiqin Chen and Hao Zhang. 2019. Learning implicit fields for generative shape modeling. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 5939–5948.Google ScholarCross Ref
7. Haoqiang Fan, Hao Su, and Leonidas J Guibas. 2017. A point set generation network for 3D object reconstruction from a single image. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 605–613.Google ScholarCross Ref
8. Lin Gao, Yu-Kun Lai, Jie Yang, Ling-Xiao Zhang, Shihong Xia, and Leif Kobbelt. 2019. Sparse Data Driven Mesh Deformation. IEEE Trans. Vis. Comput. Graph. (2019).Google ScholarCross Ref
9. Lin Gao, Jie Yang, Yi-Ling Qiao, Yu-Kun Lai, Paul L. Rosin, Weiwei Xu, and Shihong Xia. 2018. Automatic unpaired shape deformation transfer. ACM Trans. Graph. 37, 6 (2018), 237:1–237:15.Google ScholarDigital Library
10. Rohit Girdhar, David F Fouhey, Mikel Rodriguez, and Abhinav Gupta. 2016. Learning a predictable and generative vector representation for objects. In European Conference on Computer Vision (ECCV). Springer, 484–499.Google ScholarCross Ref
11. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in neural information processing systems (NIPS). 2672–2680.Google Scholar
12. Thibault Groueix, Matthew Fisher, Vladimir G. Kim, Bryan Russell, and Mathieu Aubry. 2018. AtlasNet: A Papier-Mâché Approach to Learning 3D Surface Generation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
13. Heli Ben Hamu, Haggai Maron, Itay Kezurer, Gal Avineri, and Yaron Lipman. 2018. Multi-chart generative surface modeling. ACM Trans. Graph. 37, 6 (2018), 215:1–215:15.Google Scholar
14. Rana Hanocka, Amir Hertz, Noa Fish, Raja Giryes, Shachar Fleishman, and Daniel Cohen-Or. 2019. MeshCNN: A Network with an Edge. ACM Trans. Graph. 38, 4 (2019), 90:1–90:12.Google ScholarDigital Library
15. Kenneth Holmström and Marcus M Edvall. 2004. The TOMLAB optimization environment. In Modeling Languages in Mathematical Optimization. Springer, 369–376.Google Scholar
16. Haibin Huang, Evangelos Kalogerakis, and Benjamin Marlin. 2015. Analysis and synthesis of 3D shape families via deep-learned generative models of surfaces. Comp. Graph. Forum 34, 5 (2015), 25–38.Google ScholarCross Ref
17. Jingwei Huang, Hao Su, and Leonidas Guibas. 2018. Robust watertight manifold surface generation method for ShapeNet models. arXiv:1802.01698 (2018).Google Scholar
18. Shi-Sheng Huang, Hongbo Fu, Ling-Yu Wei, and Shi-Min Hu. 2016. Support Substructures: Support-Induced Part-Level Structural Representation. IEEE Trans. Vis. Comput. Graph. 22, 8 (2016), 2024–2036.Google ScholarDigital Library
19. Dominic Jack, Jhony K Pontes, Sridha Sridharan, Clinton Fookes, Sareh Shirazi, Frederic Maire, and Anders Eriksson. 2018. Learning free-form deformations for 3d object reconstruction. In Asian Conference on Computer Vision. Springer, 317–333.Google Scholar
20. Vladimir G. Kim, Wilmot Li, Niloy J. Mitra, Siddhartha Chaudhuri, Stephen DiVerdi, and Thomas Funkhouser. 2013. Learning Part-based Templates from Large Collections of 3D Shapes. ACM Trans. Graph. 32, 4 (2013), 70:1–70:12.Google ScholarDigital Library
21. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations (ICLR).Google Scholar
22. Diederik P Kingma and Max Welling. 2013. Auto-encoding variational bayes. arXiv:1312.6114 (2013).Google Scholar
23. Jun Li, Kai Xu, Siddhartha Chaudhuri, Ersin Yumer, Hao Zhang, and Leonidas J. Guibas. 2017. GRASS: Generative Recursive Autoencoders for Shape Structures. ACM Trans. Graph. 36, 4 (2017), 52:1–52:14.Google ScholarDigital Library
24. Haggai Maron, Meirav Galun, Noam Aigerman, Miri Trope, Nadav Dym, Ersin Yumer, Vladimir G Kim, and Yaron Lipman. 2017. Convolutional neural networks on surfaces via seamless toric covers. ACM Trans. Graph. 36, 4 (2017), 71:1–71:10.Google ScholarDigital Library
25. Hsien-Yu Meng, Lin Gao, Yu-Kun Lai, and Dinesh Manocha. 2019. VV-Net: Voxel VAE Net with Group Convolutions for Point Cloud Segmentation. In IEEE International Conference on Computer Vision (ICCV).Google Scholar
26. Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. 2019. Occupancy networks: Learning 3d reconstruction in function space. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 4460–4470.Google ScholarCross Ref
27. Niloy Mitra, Michael Wand, Hao Zhang, Daniel Cohen-Or, Vladimir Kim, and Qi-Xing Huang. 2013. Structure-aware Shape Processing. In SIGGRAPH Asia 2013 Courses. 1:1–1:20.Google Scholar
28. Kaichun Mo, Paul Guerrero, Li Yi, Hao Su, Peter Wonka, Niloy Mitra, and Leonidas Guibas. 2019a. StructureNet: Hierarchical Graph Networks for 3D Shape Generation. ACM Trans. Graph. 38, 6 (2019), 242:1–242:29.Google ScholarDigital Library
29. Kaichun Mo, Shilin Zhu, Angel X Chang, Li Yi, Subarna Tripathi, Leonidas J Guibas, and Hao Su. 2019b. PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 909–918.Google ScholarCross Ref
30. Charlie Nash and Christopher KI Williams. 2017. The shape variational autoencoder: A deep generative model of part-segmented 3D objects. Comp. Graph. Forum 36, 5 (2017), 1–12.Google ScholarDigital Library
31. Maks Ovsjanikov, Wilmot Li, Leonidas Guibas, and Niloy J. Mitra. 2011. Exploration of Continuous Variability in Collections of 3D Shapes. ACM Trans. Graph. 30, 4 (2011), 33:1–33:10.Google ScholarDigital Library
32. Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Love-grove. 2019. DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 165–174.Google Scholar
33. Joshua Podolak, Philip Shilane, Aleksey Golovinskiy, Szymon Rusinkiewicz, and Thomas Funkhouser. 2006. A planar-reflective symmetry transform for 3D shapes. ACM Trans. Graph. 25, 3 (2006), 549–559.Google ScholarDigital Library
34. Gerard Pons-Moll, Javier Romero, Naureen Mahmood, and Michael J. Black. 2015. Dyna: A model of dynamic human shape in motion. ACM Trans. Graph. 34, 4 (2015), 120:1–120:14.Google ScholarDigital Library
35. Adrien Poulenard and Maks Ovsjanikov. 2018. Multi-directional Geodesic Neural Networks via Equivariant Convolution. ACM Trans. Graph. 37, 6 (2018), 236:1–236:14.Google ScholarDigital Library
36. Charles Ruizhongtai Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. 2017a. PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation.. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 77–85.Google Scholar
37. Charles Ruizhongtai Qi, Hao Su, Matthias Nießner, Angela Dai, Mengyuan Yan, and Leonidas J. Guibas. 2016. Volumetric and Multi-view CNNs for Object Classification on 3D Data. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 5648–5656.Google Scholar
38. Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J. Guibas. 2017b. PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space.. In Advances in Neural Information Processing Systems (NIPS). 5105–5114.Google Scholar
39. Ayan Sinha, Jing Bai, and Karthik Ramani. 2016. Deep Learning 3D Shape Surfaces Using Geometry Images. In European Conference on Computer Vision (ECCV). Springer, 223–240.Google ScholarCross Ref
40. Amir Arsalan Soltani, Haibin Huang, Jiajun Wu, Tejas D Kulkarni, and Joshua B Tenenbaum. 2017. Synthesizing 3D Shapes via Modeling Multi-View Depth Maps and Silhouettes with Deep Generative Networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 1511–1519.Google ScholarCross Ref
41. Olga Sorkine and Marc Alexa. 2007. As-Rigid-As-Possible Surface Modeling. In Proceedings of EUROGRAPHICS/ACM SIGGRAPH Symposium on Geometry Processing. 109–116.Google ScholarDigital Library
42. Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned-Miller. 2015. Multiview convolutional neural networks for 3D shape recognition. In IEEE International Conference on Computer Vision (ICCV). 945–953.Google Scholar
43. Qingyang Tan, Lin Gao, Yu-Kun Lai, and Shihong Xia. 2018. Variational Autoencoders for Deforming 3D Mesh Models. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 5841–5850.Google ScholarCross Ref
44. Maxim Tatarchenko, Alexey Dosovitskiy, and Thomas Brox. 2017. Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs. In IEEE International Conference on Computer Vision (ICCV). 2088–2096.Google ScholarCross Ref
45. Hao Wang, Nadav Schor, Ruizhen Hu, Haibin Huang, Daniel Cohen-Or, and Hui Huang. 2018a. Global-to-local Generative Model for 3D Shapes. ACM Trans. Graph. 37, 6 (2018), 214:1–214:10.Google ScholarDigital Library
46. Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. 2018c. Pixel2mesh: Generating 3d mesh models from single rgb images. In European Conference on Computer Vision (ECCV). Springer, 52–67.Google ScholarCross Ref
47. Peng-Shuai Wang, Yang Liu, Yu-Xiao Guo, Chun-Yu Sun, and Xin Tong. 2017. O-CNN: Octree-based Convolutional Neural Networks for 3D Shape Analysis. ACM Trans. Graph. 36, 4 (2017), 72:1–72:11.Google ScholarDigital Library
48. Peng-Shuai Wang, Chun-Yu Sun, Yang Liu, and Xin Tong. 2018b. Adaptive O-CNN: A Patch-based Deep Representation of 3D Shapes. ACM Trans. Graph. 37, 6 (2018), 217:1–217:11.Google ScholarDigital Library
49. Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Josh Tenenbaum. 2016. Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling.. In Advances in Neural Information Processing Systems (NIPS). 82–90.Google Scholar
50. Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 2015. 3D ShapeNets: A deep representation for volumetric shapes. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 1912–1920.Google Scholar
51. Zhijie Wu, Xiang Wang, Di Lin, Dani Lischinski, Daniel Cohen-Or, and Hui Huang. 2019. SAGNet: Structure-aware Generative Network for 3D-Shape Modeling. ACM Trans. Graph. 38, 4 (2019), 91:1–91:14.Google ScholarDigital Library
52. Chen Xin, Yuwei Li, Xi Luo, Tianjia Shao, Jingyi Yu, Kun Zhou, and Youyi Zheng. 2018. AutoSweep: Recovering 3D Editable Objects from a Single Photograph. IEEE Trans. Vis. Comput. Graph. (2018).Google Scholar
53. Sheng Yang, Jie Xu, Kang Chen, and Hongbo Fu. 2017. View suggestion for interactive segmentation of indoor scenes. Computational Visual Media 3, 2 (2017), 131–146.Google ScholarCross Ref
54. Li Yi, Vladimir G Kim, Duygu Ceylan, I Shen, Mengyan Yan, Hao Su, Cewu Lu, Qixing Huang, Alla Sheffer, Leonidas Guibas, et al. 2016. A scalable active framework for region annotation in 3d shape collections. ACM Trans. Graph. 35, 6 (2016), 210:1–210:12.Google ScholarDigital Library
55. Kangxue Yin, Hui Huang, Daniel Cohen-Or, and Hao Zhang. 2018. P2P-NET: Bidirectional Point Displacement Net for Shape Transform. ACM Trans. Graph. 37, 4 (2018), 152:1–152:13.Google ScholarDigital Library
56. Kun Zhou, John Synder, Baining Guo, and Heung-Yeung Shum. 2004. Iso-charts: stretch-driven mesh parameterization using spectral analysis. In Proceedings of the 2004 Eurographics/ACM SIGGRAPH symposium on Geometry processing. 45–54.Google ScholarDigital Library
57. Michael Zollhöfer, Matthias Nießner, Shahram Izadi, Christoph Rehmann, Christopher Zach, Matthew Fisher, Chenglei Wu, Andrew Fitzgibbon, Charles Loop, Christian Theobalt, and Marc Stamminger. 2014. Real-time Non-rigid Reconstruction Using an RGB-D Camera. ACM Trans. Graph. 33, 4 (2014), 156:1–156:12.Google ScholarDigital Library
