“Explorable Mesh Deformation Subspaces from Unstructured 3D Generative Models” by Maesumi, Guerrero, Kim, Fisher, Chaudhuri, et al. …
Conference:
Type(s):
Title:
- Explorable Mesh Deformation Subspaces from Unstructured 3D Generative Models
Session/Category Title: Navigating Shape Spaces
Presenter(s)/Author(s):
Abstract:
Exploring variations of 3D shapes is a time-consuming process in traditional 3D modeling tools. Deep generative models of 3D shapes often feature continuous latent spaces that can, in principle, be used to explore potential variations starting from a set of input shapes; in practice, doing so can be problematic—latent spaces are high dimensional and hard to visualize, contain shapes that are not relevant to the input shapes, and linear paths through them often lead to sub-optimal shape transitions. Furthermore, one would ideally be able to explore variations in the original high-quality meshes used to train the generative model, not its lower-quality output geometry. In this paper, we present a method to explore variations among a given set of landmark shapes by constructing a mapping from an easily-navigable 2d exploration space to a subspace of a pre-trained generative model. We first describe how to find a mapping that spans the set of input landmark shapes and exhibits smooth variations between them. We then show how to turn the variations in this subspace into deformation fields, to transfer those variations to high-quality meshes for the landmark shapes. Our results show that our method can produce visually-pleasing and easily-navigable 2D exploration spaces for several different shape categories, especially as compared to prior work on learning deformation spaces for 3D shapes.
References:
[1]
Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas J. Guibas. 2017. Learning Representations and Generative Models for 3D Point Clouds. In International Conference on Machine Learning.
[2]
Noam Aigerman, Kunal Gupta, Vladimir Kim, Siddhartha Chaudhuri, Jun Saito, and Thibault Groueix. 2022. Neural Jacobian Fields: Learning Intrinsic Mappings of Arbitrary Meshes. ACM Transactions on Graphics (Proc. SIGGRAPH) 41, 4 (2022).
[3]
Ken Anjyo, J. P. Lewis, and Frédéric Pighin. 2014. Scattered Data Interpolation for Computer Graphics. In ACM SIGGRAPH 2014 Courses (Vancouver, Canada) (SIGGRAPH ’14). Association for Computing Machinery, New York, NY, USA, Article 27, 69 pages. https://doi.org/10.1145/2614028.2615425
[4]
Melinos Averkiou, Vladimir G. Kim, Youyi Zheng, and Niloy J. Mitra. 2014. ShapeSynth: Parameterizing Model Collections for Coupled Shape Exploration and Synthesis. Computer Graphics Forum (Proc. of Eurographics) 33, 2 (2014).
[5]
Mario Botsch and Olga Sorkine. 2008. On linear variational surface deformation methods. IEEE Transactions on Visualization and Computer Graphics 14, 1 (2008), 213–230.
[6]
Angel X. Chang, Thomas A. Funkhouser, Leonidas J. Guibas, Pat Hanrahan, Qi-Xing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. 2015. ShapeNet: An Information-Rich 3D Model Repository. CoRR abs/1512.03012 (2015). arXiv:1512.03012http://arxiv.org/abs/1512.03012
[7]
Siddhartha Chaudhuri, Evangelos Kalogerakis, Stephen Giguere, and Thomas Funkhouser. 2013. AttribIt: Content Creation with Semantic Attributes. 193–202.
[8]
Siddhartha Chaudhuri, Evangelos Kalogerakis, Leonidas Guibas, and Vladlen Koltun. 2011. Probabilistic Reasoning for Assembly-Based 3D Modeling. ACM Transactions on Graphics 30, 4 (2011), 35.
[9]
Zhiqin Chen and Hao Zhang. 2019a. Learning Implicit Fields for Generative Shape Modeling. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2019).
[10]
Zhiqin Chen and Hao Zhang. 2019b. Learning Implicit Fields for Generative Shape Modeling. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2019).
[11]
Nadav Dym, Haggai Maron, and Yaron Lipman. 2017. DS++: A Flexible, Scalable and Provably Tight Relaxation for Matching Problems. In ACM SIGGRAPH Asia.
[12]
Ohad Fried, Stephen DiVerdi, Maciej Halber, Elena Sizikova, and Adam Finkelstein. 2015. IsoMatch: Creating Informative Grid Layouts. Computer Graphics Forum (Proc. Eurographics) 34, 2 (May 2015).
[13]
Lin Gao, Jie Yang, Tong Wu, Yu-Jie Yuan, Hongbo Fu, Yu-Kun Lai, and Hao (Richard) Zhang. 2019. SDM-NET: Deep Generative Network for Structured Deformable Mesh.
[14]
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, Günter Klambauer, and Sepp Hochreiter. 2017. GANs Trained by a Two Time-Scale Update Rule Converge to a Nash Equilibrium. CoRR abs/1706.08500 (2017). arXiv:1706.08500http://arxiv.org/abs/1706.08500
[15]
Jingwei Huang, Hao Su, and Leonidas Guibas. 2018. Robust Watertight Manifold Surface Generation Method for ShapeNet Models. arXiv preprint arXiv:1802.01698 (2018).
[16]
Qixing Huang, Xiangru Huang, Bo Sun, Zaiwei Zhang, Junfeng Jiang, and Chandrajit Bajaj. 2021. ARAPReg: An As-Rigid-As Possible Regularization Loss for Learning Deformable Shape Generators. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV). 5795–5805. https://doi.org/10.1109/ICCV48922.2021.00576
[17]
K. Hui, R. Li, J. Hu, and C. Fu. 2022a. Neural Template: Topology-aware Reconstruction and Disentangled Generation of 3D Meshes. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE Computer Society. https://doi.org/10.1109/CVPR52688.2022.01802
[18]
Ka-Hei Hui, Ruihui Li, Jingyu Hu, and Chi-Wing Fu. 2022b. Neural Wavelet-Domain Diffusion for 3D Shape Generation. In SIGGRAPH Asia 2022 Conference Papers (Daegu, Republic of Korea) (SA ’22). Association for Computing Machinery, New York, NY, USA, Article 24, 9 pages. https://doi.org/10.1145/3550469.3555394
[19]
Alec Jacobson, Ilya Baran, Jovan Popović, and Olga Sorkine. 2011. Bounded Biharmonic Weights for Real-Time Deformation. ACM Transactions on Graphics (Proc. SIGGRAPH) 30, 4 (2011), 78:1–78:8.
[20]
Arjun Jain, Thorsten Thormählen, Tobias Ritschel, and Hans-Peter Seidel. 2012. Exploring Shape Variations by 3D-Model Decomposition and Part-based Recombination. Comp. Graph. Forum (Proc. Eurographics 2012) 31, 2 (2012).
[21]
Chiyu Jiang, Jingwei Huang, Andrea Tagliasacchi, and Leonidas Guibas. 2020. ShapeFlow: Learnable Deformations Among 3D Shapes. In Advances in Neural Information Processing Systems.
[22]
R. Kenny Jones, Theresa Barton, Xianghao Xu, Kai Wang, Ellen Jiang, Paul Guerrero, Niloy J. Mitra, and Daniel Ritchie. 2020. ShapeAssembly: Learning to Generate Programs for 3D Shape Structure Synthesis. ACM Transactions on Graphics 39, 6 (2020).
[23]
Evangelos Kalogerakis, Siddhartha Chaudhuri, Daphne Koller, and Vladlen Koltun. 2012. A Probabilistic Model of Component-Based Shape Synthesis. ACM Transactions on Graphics 31, 4 (2012).
[24]
H. Karcher. 1977. Riemannian center of mass and mollifier smoothing. Communications on Pure and Applied Mathematics 30, 5 (1977), 509–541. https://doi.org/10.1002/cpa.3160300502 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/cpa.3160300502
[25]
Tero Karras, Samuli Laine, and Timo Aila. 2018. A Style-Based Generator Architecture for Generative Adversarial Networks. CoRR abs/1812.04948 (2018). arXiv:1812.04948http://arxiv.org/abs/1812.04948
[26]
Vladimir G. Kim, Wilmot Li, Niloy J. Mitra, Siddhartha Chaudhuri, Stephen DiVerdi, and Thomas Funkhouser. 2013. Learning Part-based Templates from Large Collections of 3D Shapes. Transactions on Graphics (Proc. of SIGGRAPH) 32, 4 (2013).
[27]
Vladimir G. Kim, Wilmot Li, Niloy J. Mitra, Stephen DiVerdi, and Thomas Funkhouser. 2012. Exploring Collections of 3D Models using Fuzzy Correspondences. Transactions on Graphics (Proc. of SIGGRAPH) 31, 4 (2012).
[28]
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
[29]
Yanir Kleiman, Joel Lanir, Dov Danon, Yasmin Felberbaum, and Daniel Cohen-Or. 2015. DynamicMaps: Similarity-Based Browsing through a Massive Set of Images. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems(CHI ’15). Association for Computing Machinery, 995–1004.
[30]
Samuli Laine. 2018. Feature-Based Metrics for Exploring the Latent Space of Generative Models. https://openreview.net/forum?id=BJslDBkwG
[31]
Jun Li, Kai Xu, Siddhartha Chaudhuri, Ersin Yumer, Hao Zhang, and Leonidas Guibas. 2017. GRASS: Generative recursive autoencoders for shape structures. ACM Transactions on Graphics 36, 4 (2017), 52.
[32]
Ruihui Li, Xianzhi Li, Ke-Hei Hui, and Chi-Wing Fu. 2021. SP-GAN:Sphere-Guided 3D Shape Generation and Manipulation. ACM Transactions on Graphics (Proc. SIGGRAPH) 40, 4 (2021).
[33]
Minghua Liu, Minhyuk Sung, Radomir Mech, and Hao Su. 2021. DeepMetaHandles: Learning Deformation Meta-Handles of 3D Meshes with Biharmonic Coordinates. arXiv preprint arXiv:2102.09105 (2021).
[34]
Joe Marks, Brad Andalman, Paul A Beardsley, William Freeman, Sarah Gibson, Jessica Hodgins, Thomas Kang, Brian Mirtich, Hanspeter Pfister, Wheeler Ruml, 1997. Design galleries: A general approach to setting parameters for computer graphics and animation. In Proceedings of the 24th annual conference on Computer graphics and interactive techniques. ACM Press/Addison-Wesley Publishing Co., 389–400.
[35]
Leland McInnes, John Healy, and James Melville. 2020. UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction. arxiv:1802.03426 [stat.ML]
[36]
Paritosh Mittal, Yen-Chi Cheng, Maneesh Singh, and Shubham Tulsiani. 2022. AutoSDF: Shape Priors for 3D Completion, Reconstruction and Generation. In CVPR.
[37]
Sanjeev Muralikrishnan, Siddhartha Chaudhuri, Noam Aigerman, Vladimir Kim, Matthew Fisher, and Niloy Mitra. 2022. GLASS: Geometric Latent Augmentation for Shape Spaces. In CVPR.
[38]
Charlie Nash, Yaroslav Ganin, SM Ali Eslami, and Peter Battaglia. 2020. Polygen: An autoregressive generative model of 3d meshes. In International Conference on Machine Learning. PMLR, 7220–7229.
[39]
Maks Ovsjanikov, Wilmot Li, Leonidas Guibas, and Niloy Mitra. 2011. Exploration of Continuous Variability in Collections of 3D Shapes. ACM Transactions on Graphics (Proc. SIGGRAPH) (2011).
[40]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. CoRR abs/1912.01703 (2019). arXiv:1912.01703http://arxiv.org/abs/1912.01703
[41]
Songyou Peng, Chiyu “Max” Jiang, Yiyi Liao, Michael Niemeyer, Marc Pollefeys, and Andreas Geiger. 2021. Shape As Points: A Differentiable Poisson Solver. In Advances in Neural Information Processing Systems (NeurIPS).
[42]
Charles R Qi, Li Yi, Hao Su, and Leonidas J Guibas. 2017. PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space. arXiv preprint arXiv:1706.02413 (2017).
[43]
Raghad Rowaida. 2021. Exploration of Latent Spaces of 3D Shapes via Isomap and Inverse Mapping. Master’s thesis. Carleton University.
[44]
Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. FaceNet: A Unified Embedding for Face Recognition and Clustering. CoRR abs/1503.03832 (2015). arXiv:1503.03832http://arxiv.org/abs/1503.03832
[45]
Olga Sorkine and Marc Alexa. 2007. As-Rigid-As-Possible Surface Modeling. Symposium on Geometry Processing, 109–116. https://doi.org/10.1145/1281991.1282006
[46]
Jerry O. Talton, Daniel Gibson, Lingfeng Yang, Pat Hanrahan, and Vladlen Koltun. 2009. Exploratory Modeling with Collaborative Design Spaces. 28, 5 (2009).
[47]
user147263. 2014. You cant solve Laplaces equation with boundary conditions on isolated points. But why? (2014). https://math.stackexchange.com/q/1031377
[48]
Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing Data using t-SNE. Journal of Machine Learning Research 9, 86 (2008), 2579–2605. http://jmlr.org/papers/v9/vandermaaten08a.html
[49]
Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. 2018. Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images. In ECCV.
[50]
Jiajun Wu, Chengkai Zhang, Tianfan Xue, William T Freeman, and Joshua B Tenenbaum. 2016. Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling. In Advances In Neural Information Processing Systems. 82–90.
[51]
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. 2022. Tackling the Generative Learning Trilemma with Denoising Diffusion GANs. In International Conference on Learning Representations (ICLR).
[52]
Zhan Xu, Yang Zhou, Li Yi, and Evangelos Kalogerakis. 2022. Morig: Motion-Aware Rigging of Character Meshes from Point Clouds. ACM Transactions on Graphics (Proc. SIGGRAPH Asia) 41, 6 (2022).
[53]
Guandao Yang, Xun Huang, Zekun Hao, Ming-Yu Liu, Serge Belongie, and Bharath Hariharan. 2019. PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flows. arXiv (2019).
[54]
Jie Yang, Kaichun Mo, Yu-Kun Lai, Leonidas J Guibas, and Lin Gao. 2020. DSM-Net: Disentangled Structured Mesh Net for Controllable Generation of Fine Geometry. arXiv preprint arXiv:2008.05440 (2020).
[55]
Wang Yifan, Noam Aigerman, Vladimir G. Kim, Siddhartha Chaudhuri, and Olga Sorkine-Hornung. 2020. Neural Cages for Detail-Preserving 3D Deformations. In CVPR.
[56]
Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojcic, Or Litany, Sanja Fidler, and Karsten Kreis. 2022. LION: Latent Point Diffusion Models for 3D Shape Generation. In Advances in Neural Information Processing Systems (NeurIPS).
[57]
Xin-Yang Zheng, Yang Liu, Peng-Shuai Wang, and Xin Tong. 2022. SDF-StyleGAN: Implicit SDF-Based StyleGAN for 3D Shape Generation. In Comput. Graph. Forum (SGP).
