“GEM3D: GEnerative Medial Abstractions for 3D Shape Synthesis”
Conference:
Type(s):
Title:
- GEM3D: GEnerative Medial Abstractions for 3D Shape Synthesis
Presenter(s)/Author(s):
Abstract:
We introduce GEM3D ? a deep, topology-aware model for generating and reconstructing 3D shapes. Our key ingredient is a neural skeleton-based representation compactly encoding both shape topology and geometry. Experiments show significantly more faithful surface reconstruction and diverse shape generation compared to prior work, especially for structurally complex shapes.
References:
[1]
P. Achlioptas, O. Diamanti, I. Mitliagkas, and L.J. Guibas. 2018a. Learning representations and generative models for 3D point clouds. In Proc. ICML.
[2]
Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. 2018b. Learning Representations and Generative Models for 3D Point Clouds.
[3]
Nina Amenta and Marshall Bern. 1998. Surface reconstruction by Voronoi filtering. In Proceedings of the Fourteenth Annual Symposium on Computational Geometry.
[4]
Jules Bloomenthal and Ken Shoemake. 1991. Convolution surfaces. Proc. ACM SIGGRAPH, 251?256.
[5]
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, 2015. Shapenet: An information-rich 3D model repository. arXiv preprint arXiv:1512.03012 (2015).
[6]
Siddhartha Chaudhuri, Daniel Ritchie, Jiajun Wu, Kai Xu, and Hao (Richard) Zhang. 2020. Learning Generative Models of 3D Structures. In Computer Graphics Forum.
[7]
Zhiqin Chen and Hao Zhang. 2019. Learning Implicit Fields for Generative Shape Modeling. CVPR (2019).
[8]
Yen-Chi Cheng, Hsin-Ying Lee, Sergey Tulyakov, Alexander G Schwing, and Liang-Yan Gui. 2023. SDFusion: Multimodal 3d shape completion, reconstruction, and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 4456?4465.
[9]
Matt?o Cl?mot and Julie Digne. 2023. Neural skeleton: Implicit neural representation away from the surface. Computers & Graphics 114 (2023), 368?378.
[10]
Angela Dai and Matthias Nie?ner. 2019. Scan2Mesh: From Unstructured Range Scans to 3D Meshes. In Proc. CVPR.
[11]
Zhiyang Dou, Cheng Lin, Rui Xu, Lei Yang, Shiqing Xin, Taku Komura, and Wenping Wang. 2022. Coverage axis: Inner point selection for 3d shape skeletonization. In Computer Graphics Forum, Vol. 41. Wiley Online Library, 419?432.
[12]
Anastasia Dubrovina, Fei Xia, Panos Achlioptas, Mira Shalah, Rapha?l Groscot, and Leonidas J Guibas. 2019. Composite shape modeling via latent space factorization. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 8140?8149.
[13]
Matthew Fisher, Manolis Savva, and Pat Hanrahan. 2011. Characterizing structural relationships in scenes using graph kernels. In ACM Transactions on Graphics, Vol. 30. ACM, 34.
[14]
Mengyuan Ge, Junfeng Yao, Baorong Yang, Ningna Wang, Zhonggui Chen, and Xiaohu Guo. 2023. Point2MM: Learning medial mesh from point clouds. Computers & Graphics 115 (2023), 511?521.
[15]
Minghao Guo, Bohan Wang, and Wojciech Matusik. 2023. Medial Skeletal Diagram: A Generalized Medial Axis Approach for Compact 3D Shape Representation. arxiv:2310.09395
[16]
Masaki Hilaga, Yoshihisa Shinagawa, Taku Kohmura, and Tosiyasu L. Kunii. 2001. Topology matching for fully automatic similarity estimation of 3D shapes. In Proc. SIGGRAPH.
[17]
Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising Diffusion Probabilistic Models. In Proc. NIPS.
[18]
Jianwei Hu, Gang Chen, Baorong Yang, Ningna Wang, Xiaohu Guo, and Bin Wang. 2022. IMMAT: Mesh reconstruction from single view images by medial axis transform prediction. Computer-Aided Design 150 (2022), 103304.
[19]
Jianwei Hu, Ningna Wang, Baorong Yang, Gang Chen, Xiaohu Guo, and Bin Wang. 2023. S3DS: Self-supervised Learning of 3D Skeletons from Single View Images. In Proceedings of the 31st ACM International Conference on Multimedia. 6948?6958.
[20]
Haibin Huang, Evangelos Kalogerakis, and Benjamin Marlin. 2015. Analysis and synthesis of 3D shape families via deep-learned generative models of surfaces. Computer Graphics Forum 34, 5 (2015).
[21]
Ka-Hei Hui, Ruihui Li, Jingyu Hu, and Chi-Wing Fu. 2022. Neural Wavelet-domain Diffusion for 3D Shape Generation. In Proc. ACM SIGGRAPH ASIA.
[22]
Evangelos Kalogerakis, Siddhartha Chaudhuri, Daphne Koller, and Vladlen Koltun. 2012. A Probabilistic Model for Component-Based Shape Synthesis. In Proc. ACM SIGGRAPH.
[23]
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. 2022. Elucidating the Design Space of Diffusion-Based Generative Models. In Proc. NIPS.
[24]
Diederik P. Kingma and Max Welling. 2014. Auto-Encoding Variational Bayes. In Proc. ICLR.
[25]
Lei Lan, Ran Luo, Marco Fratarcangeli, Weiwei Xu, Huamin Wang, Xiaohu Guo, Junfeng Yao, and Yin Yang. 2020. Medial elastics: Efficient and collision-ready deformation via medial axis transform. ACM Transactions on Graphics (TOG) 39, 3 (2020), 1?17.
[26]
Jun Li, Kai Xu, Siddhartha Chaudhuri, Ersin Yumer, Hao Zhang, and Leonidas Guibas. 2017. GRASS: Generative Recursive Autoencoders for Shape Structures. Trans. Graphics 36, 4 (2017).
[27]
Pan Li, Bin Wang, Feng Sun, Xiaohu Guo, Caiming Zhang, and Wenping Wang. 2016. Q-MAT: Computing Medial Axis Transform By Quadratic Error Minimization. ACM Transactions on Graphics 35, 1 (2016).
[28]
Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin. 2023. Magic3D: High-Resolution Text-to-3D Content Creation. In Proc. CVPR.
[29]
J. Liu, F. Yu, and T. Funkhouser. 2017. Interactive 3D modeling with a generative adversarial network. In Proc. 3DV.
[30]
William E Lorensen and Harvey E Cline. 1987. Marching cubes: A high resolution 3D surface construction algorithm. ACM Transactions on Graphics 21, 4 (1987).
[31]
Ling Luo, Pinaki Nath Chowdhury, Tao Xiang, Yi-Zhe Song, and Yulia Gryaditskaya. 2023. 3D VR Sketch Guided 3D Shape Prototyping and Exploration. In Proceedings of the IEEE/CVF International Conference on Computer Vision.
[32]
Paul Merrell, Eric Schkufza, and Vladlen Koltun. 2010. Computer-Generated Residential Building Layouts. In ACM Transactions on Graphics.
[33]
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. 2019. Occupancy networks: Learning 3D reconstruction in function space. In Proc. CVPR.
[34]
Przemyslaw Musialski, Peter Wonka, Daniel G. Aliaga, Michael Wimmer, Luc van Gool, and Werner Purgathofer. 2013. A Survey of Urban Reconstruction. In Computer Graphics Forum.
[35]
Yinyu Nie, Yiqun Lin, Xiaoguang Han, Shihui Guo, Jian Chang, Shuguang Cui, and Jian Jun Zhang. 2020. Skeleton-bridged point completion: from global inference to local adjustment. In Proc. NIPS.
[36]
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. 2019. Deepsdf: Learning continuous signed distance functions for shape representation. In Proc. CVPR.
[37]
Akshay Gadi Patil, Supriya Gadi Patil, Manyi Li, Matthew Fisher, Manolis Savva, and Hao Zhang. 2020. Advances in Data-Driven Analysis and Synthesis of 3D Indoor Scenes. In Computer Graphics Forum.
[38]
Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger. 2020. Convolutional occupancy networks. In Proc. ECCV.
[39]
Dmitry Petrov, Matheus Gadelha, Radom?r M?ch, and Evangelos Kalogerakis. 2023. ANISE: Assembly-based Neural Implicit Surface rEconstruction. IEEE Transactions on Visualization and Computer Graphics (2023).
[40]
Pixologic. 2022. ZBrush. https://www.maxon.net/en/zbrush
[41]
Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. 2023. DreamFusion: Text-to-3D using 2D Diffusion. ICLR (2023).
[42]
Przemyslaw Prusinkiewicz and Aristid Lindenmayer. 1990. The Algorithmic Beauty of Plants.
[43]
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. 2017. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In Proc. NIPS.
[44]
Daniel Rebain, Ke Li, Vincent Sitzmann, Soroosh Yazdani, Kwang Moo Yi, and Andrea Tagliasacchi. 2021. Deep medial fields. arXiv preprint arXiv:2106.03804 (2021).
[45]
Mehdi S. M. Sajjadi, Olivier Bachem, Mario Lu?i?, Olivier Bousquet, and Sylvain Gelly. 2018. Assessing Generative Models via Precision and Recall. In Advances in Neural Information Processing Systems (NeurIPS).
[46]
Zifan Shi, Sida Peng, Yinghao Xu, Andreas Geiger, Yiyi Liao, and Yujun Shen. 2023. Deep Generative Models on 3D Representations: A Survey. arxiv:2210.15663 [cs.CV]
[47]
J Ryan Shue, Eric Ryan Chan, Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein. 2023. 3D neural field generation using triplane diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 20875?20886.
[48]
Vincent Sitzmann, Julien N.P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. 2020. Implicit Neural Representations with Periodic Activation Functions. In Proc. NIPS.
[49]
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. 2021. Score-Based Generative Modeling through Stochastic Differential Equations. In Proc. ICLR.
[50]
A. Tagliasacchi, I. Alhashim, M. Olson, and H. Zhang. 2012. Mean Curvature Skeletons. 31, 5 (2012).
[51]
Andrea Tagliasacchi, Thomas Delame, Michela Spagnuolo, Nina Amenta, and Alexandru Telea. 2016. 3D Skeletons: A State-of-the-Art Report. Computer Graphics Forum 35, 2 (2016).
[52]
Andrea Tagliasacchi, Hao Zhang, and Daniel Cohen-Or. 2009. Curve skeleton extraction from incomplete point cloud. ACM Transactions on Graphics 28 (2009).
[53]
Jiapeng Tang, Xiaoguang Han, Junyi Pan, Kui Jia, and Xin Tong. 2019. A skeleton-bridged deep learning approach for generating meshes of complex topologies from single rgb images. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition. 4541?4550.
[54]
Jiapeng Tang, Xiaoguang Han, Mingkui Tan, Xin Tong, and Kui Jia. 2021. SkeletonNet: A topology-preserving solution for learning mesh reconstruction of object surfaces from rgb images. IEEE Transactions on Pattern Analysis and Machine Intelligence 44, 10 (2021), 6454?6471.
[55]
Jean-Marc Thiery, ?milie Guy, and Tamy Boubekeur. 2013. Sphere-Meshes: shape approximation using spherical quadric error metrics. ACM Transactions on Graphics 32, 6 (2013).
[56]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proc. NIPS.
[57]
Ningna Wang, Bin Wang, Wenping Wang, and Xiaohu Guo. 2022. Computing medial axis transform with feature preservation via restricted power diagram. ACM Transactions on Graphics (TOG) 41, 6 (2022), 1?18.
[58]
Rundi Wu, Yixin Zhuang, Kai Xu, Hao Zhang, and Baoquan Chen. 2020. PQ-Net: A generative part seq2seq network for 3d shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 829?838.
[59]
Shihao Wu, Hui Huang, Minglun Gong, Matthias Zwicker, and Daniel Cohen-Or. 2015. Deep points consolidation. ACM Transactions on Graphics 34 (2015).
[60]
Qun-Ce Xu, Tai-Jiang Mu, and Yong-Liang Yang. 2023. A survey of deep learning-based 3D shape generation. Computational Visual Media 9 (2023), 407?442. Issue 3.
[61]
Zhan Xu, Yang Zhou, Li Yi, and Evangelos Kalogerakis. 2022a. MoRig: Motion-Aware Rigging of Character Meshes from Point Clouds. In Proc. ACM SIGGRAPH ASIA.
[62]
Zhan Xu, Matthew Fisher, Yang Zhou, Deepali Aneja, Rushikesh Dudhat, Li Yi, and Evangelos Kalogerakis. 2022b. APES: Articulated Part Extraction from Sprite Sheets. In Proc. CVPR.
[63]
Zhan Xu, Yang Zhou, Evangelos Kalogerakis, Chris Landreth, and Karan Singh. 2020. RigNet: Neural Rigging for Articulated Characters. ACM Trans. on Graphics 39 (2020).
[64]
Zhan Xu, Yang Zhou, Evangelos Kalogerakis, and Karan Singh. 2019. Predicting Animation Skeletons for 3D Articulated Models via Volumetric Nets. In 2019 International Conference on 3D Vision (3DV).
[65]
Yajie Yan, David Letscher, and Tao Ju. 2018. Voxel cores: Efficient, robust, and provably good approximation of 3d medial axes. ACM Transactions on Graphics (TOG) 37, 4 (2018), 1?13.
[66]
Baorong Yang, Junfeng Yao, Bin Wang, Jianwei Hu, Yiling Pan, Tianxiang Pan, Wenping Wang, and Xiaohu Guo. 2020. P2MAT-NET: Learning medial axis transform from sparse point clouds. Computer Aided Geometric Design 80 (2020), 101874.
[67]
Kangxue Yin, Hui Huang, Daniel Cohen-Or, and Hao Zhang. 2018. P2P-NET: Bidirectional Point Displacement Net for Shape Transform. ACM Transactions on Graphics 37, 4 (2018).
[68]
Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. 2022. Point-bert: Pre-training 3D point cloud transformers with masked point modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 19313?19322.
[69]
C. Zanni, A. Bernhardt, M. Quiblier, and M.-P. Cani. 2013. SCALe-invariant Integral Surfaces. Computer Graphics Forum (2013).
[70]
Biao Zhang, Matthias Nie?ner, and Peter Wonka. 2022. 3DILG: Irregular Latent Grids for 3D Generative Modeling. In Proc. NIPS.
[71]
Biao Zhang, Jiapeng Tang, Matthias Nie?ner, and Peter Wonka. 2023. 3DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Models. ACM Transactions on Graphics 42, 4 (2023).
[72]
Xin-Yang Zheng, Hao Pan, Peng-Shuai Wang, Xin Tong, Yang Liu, and Heung-Yeung Shum. 2023. Locally attentional sdf diffusion for controllable 3D shape generation. ACM Transactions on Graphics (TOG) 42, 4 (2023), 1?13.
[73]
Jie Zhou, Zhongjin Luo, Qian Yu, Xiaoguang Han, and Hongbo Fu. 2023. GA-Sketching: Shape Modeling from Multi-View Sketching with Geometry-Aligned Deep Implicit Functions. In Computer Graphics Forum, Vol. 42. Wiley Online Library, e14948.
[74]
Qingnan Zhou and Alec Jacobson. 2016. Thingi10K: A Dataset of 10,000 3D-Printing Models. arXiv preprint arXiv:1605.04797 (2016).
