“Neural Subdivision” by Liu, Kim, Chaudhuri, Aigerman and Jacobson
Conference:
Type(s):
Title:
- Neural Subdivision
Session/Category Title: Modeling and Synthesis
Presenter(s)/Author(s):
Abstract:
This paper introduces Neural Subdivision, a novel framework for data-driven coarse-to-fine geometry modeling. During inference, our method takes a coarse triangle mesh as input and recursively subdivides it to a finer geometry by applying the fixed topological updates of Loop Subdivision, but predicting vertex positions using a neural network conditioned on the local geometry of a patch. This approach enables us to learn complex non-linear subdivision schemes, beyond simple linear averaging used in classical techniques. One of our key contributions is a novel self-supervised training setup that only requires a set of high-resolution meshes for learning network weights. For any training shape, we stochastically generate diverse low-resolution discretizations of coarse counterparts, while maintaining a bijective mapping that prescribes the exact target position of every new vertex during the subdivision process. This leads to a very efficient and accurate loss function for conditional mesh generation, and enables us to train a method that generalizes across discretizations and favors preserving the manifold structure of the output. During training we optimize for the same set of network weights across all local mesh patches, thus providing an architecture that is not constrained to a specific input mesh, fixed genus, or category. Our network encodes patch geometry in a local frame in a rotation- and translation-invariant manner. Jointly, these design choices enable our method to generalize well, and we demonstrate that even when trained on a single high-resolution mesh our method generates reasonable subdivisions for novel shapes.
References:
1. Noam Aigerman, Roi Poranne, and Yaron Lipman. 2014. Lifted bijections for low distortion surface mappings. ACM Transactions on Graphics (TOG) 33, 4 (2014), 69.
2. Noam Aigerman, Roi Poranne, and Yaron Lipman. 2015. Seamless surface mappings. ACM Transactions on Graphics (TOG) 34, 4 (2015), 72.
3. Harry G. Barrow, Jay M. Tenenbaum, Robert C. Bolles, and Helen C. Wolf. 1977. Parametric Correspondence and Chamfer Matching: Two New Techniques for Image Matching. In Proceedings of the 5th International Joint Conference on Artificial Intelligence. Cambridge, MA, USA, August 22-25, 1977, Raj Reddy (Ed.). William Kaufmann, 659–663.
4. Alexander I. Bobenko, Helmut Pottmann, and Thilo Rörig. 2020. Multi-Nets. Classification of Discrete and Smooth Surfaces with Characteristic Properties on Arbitrary Parameter Rectangles. Discret. Comput. Geom. 63, 3 (2020), 624–655.
5. Alexander M. Bronstein, Michael M. Bronstein, and Ron Kimmel. 2009. Numerical Geometry of Non-Rigid Shapes. Springer.
6. Michael M. Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. 2017. Geometric Deep Learning: Going beyond Euclidean data. IEEE Signal Process. Mag. 34, 4 (2017), 18–42.
7. E. Catmull and J. Clark. 1998. Recursively Generated B-Spline Surfaces on Arbitrary Topological Meshes. Association for Computing Machinery, New York, NY, USA, 183–188.
8. Siddhartha Chaudhuri, Daniel Ritchie, Jiajun Wu, Kai Xu, and Hao (Richard) Zhang. 2020. Learning to Generate 3D Structures. In Eurographics State-of-the-Art Report (STAR).
9. Paolo Cignoni, Claudio Rocchini, and Roberto Scopigno. 1998. Metro: measuring error on simplified surfaces. In Computer graphics forum, Vol. 17. Wiley Online Library, 167–174.
10. Jonathan Cohen, Marc Olano, and Dinesh Manocha. 1998. Appearance-Preserving Simplification. In Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’98). Association for Computing Machinery, New York, NY, USA, 115–122.
11. Jonathan D. Cohen, Dinesh Manocha, and Marc Olano. 1997. Simplifying polygonal models using successive mappings. In IEEE Visualization ’97, Proceedings, Phoenix, AZ, USA, October 19-24, 1997. IEEE Computer Society and ACM, 395–402.
12. Jonathan D. Cohen, Dinesh Manocha, and Marc Olano. 2003. Successive Mappings: An Approach to Polygonal Mesh Simplification with Guaranteed Error Bounds. Int. J. Comput. Geometry Appl. 13, 1 (2003), 61.
13. Angela Dai and Matthias Nießner. 2019. Scan2Mesh: From Unstructured Range Scans to 3D Meshes. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019. Computer Vision Foundation / IEEE, 5574–5583.
14. Tony DeRose, Michael Kass, and Tien Truong. 1998. Subdivision Surfaces in Character Animation. In Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1998, Orlando, FL, USA, July 19-24, 1998, Steve Cunningham, Walt Bransford, and Michael F. Cohen (Eds.). ACM, 85–94.
15. Tamal K Dey, Herbert Edelsbrunner, Sumanta Guha, and Dmitry V Nekhayev. 1999. Topology preserving edge contraction. Publ. Inst. Math.(Beograd)(NS) 66, 80 (1999), 23–45.
16. Daniel Doo. 1978. A subdivision algorithm for smoothing down irregularly shaped polyhederons. Computer Aided Design (1978), 157–165.
17. D. Doo and M. Sabin. 1998. Behaviour of Recursive Division Surfaces near Extraordinary Points. Association for Computing Machinery, New York, NY, USA, 177–181.
18. Nira Dyn, David Levine, and John A. Gregory. 1990. A butterfly subdivision scheme for surface interpolation with tension control. ACM Trans. Graph. 9, 2 (1990), 160–169.
19. Haoqiang Fan, Hao Su, and Leonidas J. Guibas. 2017. A Point Set Generation Network for 3D Object Reconstruction from a Single Image. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017. IEEE Computer Society, 2463–2471.
20. Michael S. Floater and Charles A. Micchelli. 1997. Nonlinear Stationary Subdivision. Journal of Approximation Theory (1997).
21. Michael Garland and Paul S. Heckbert. 1997. Surface simplification using quadric error metrics. In Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1997, Los Angeles, CA, USA, August 3-8, 1997, G. Scott Owen, Turner Whitted, and Barbara Mones-Hattal (Eds.). ACM, 209–216.
22. Thibault Groueix, Matthew Fisher, Vladimir G. Kim, Bryan C. Russell, and Mathieu Aubry. 2018a. 3D-CODED: 3D Correspondences by Deep Deformation. In Computer Vision – ECCV 2018 – 15th European Conference, Munich, Germany, September 8-14, 2018. Proceedings, Part II (Lecture Notes in Computer Science), Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (Eds.), Vol. 11206. Springer, 235–251.
23. Thibault Groueix, Matthew Fisher, Vladimir G. Kim, Bryan C. Russell, and Mathieu Aubry. 2018b. A Papier-Mâché Approach to Learning 3D Surface Generation. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018. IEEE Computer Society, 216–224.
24. Igor Guskov, Andrei Khodakovsky, Peter Schröder, and Wim Sweldens. 2002. Hybrid meshes: multiresolution using regular and irregular refinement. In Proceedings of the 18th Annual Symposium on Computational Geometry, Barcelona, Spain, June 5-7, 2002, Ferran Hurtado, Vera Sacristán, Chandrajit Bajaj, and Subhash Suri (Eds.). ACM, 264–272.
25. Igor Guskov, Kiril Vidimce, Wim Sweldens, and Peter Schröder. 2000. Normal meshes. In Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 2000, New Orleans, LA, USA, July 23-28, 2000, Judith R. Brown and Kurt Akeley (Eds.). ACM, 95–102.
26. Rana Hanocka, Amir Hertz, Noa Fish, Raja Giryes, Shachar Fleishman, and Daniel Cohen-Or. 2019. MeshCNN: A Network with an Edge. ACM Transactions on Graphics (TOG) 38, 4 (2019), 90.
27. Hugues Hoppe, Tony DeRose, Tom Duchamp, Mark A. Halstead, Hubert Jin, John Alan McDonald, Jean Schweitzer, and Werner Stuetzle. 1994. Piecewise smooth surface reconstruction. In Proceedings of the 21th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1994, Orlando, FL, USA, July 24-29, 1994, Dino Schweitzer, Andrew S. Glassner, and Mike Keeler (Eds.). ACM, 295–302.
28. Hugues Hoppe, Tony DeRose, Tom Duchamp, John Alan McDonald, and Werner Stuetzle. 1993. Mesh optimization. In Proceedings of the 20th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1993, Anaheim, CA, USA, August 2-6, 1993, Mary C. Whitton (Ed.). ACM, 19–26.
29. Krishna Murthy J., Edward Smith, Jean-Francois Lafleche, Clement Fuji Tsang, Artem Rozantsev, Wenzheng Chen, Tommy Xiang, Rev Lebaredian, and Sanja Fidler. 2019. Kaolin: A PyTorch Library for Accelerating 3D Deep Learning Research. arXiv:1911.05063 (2019).
30. Kestutis Karciauskas and Jörg Peters. 2018. A New Class of Guided C2 Subdivision Surfaces Combining Good Shape with Nested Refinement. Comput. Graph. Forum 37, 6 (2018), 84–95.
31. Ladislav Kavan, Dan Gerszewski, Adam W. Bargteil, and Peter-Pike Sloan. 2011. Physics-Inspired Upsampling for Cloth Simulation in Games. ACM Trans. Graph. 30, 4, Article Article 93 (July 2011), 10 pages.
32. Michael M. Kazhdan and Hugues Hoppe. 2013. Screened poisson surface reconstruction. ACM Trans. Graph. 32, 3 (2013), 29:1–29:13.
33. Andrei Khodakovsky, Nathan Litke, and Peter Schröder. 2003. Globally Smooth Parameterizations with Low Distortion. ACM Trans. Graph. 22, 3 (July 2003), 350–357.
34. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, Yoshua Bengio and Yann LeCun (Eds.).
35. Leif Kobbelt. 1996. Interpolatory Subdivision on Open Quadrilateral Nets with Arbitrary Topology. Comput. Graph. Forum 15, 3 (1996), 409–420.
36. Leif Kobbelt. 2000. 3-subdivision. In Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 2000, New Orleans, LA, USA, July 23-28, 2000, Judith R. Brown and Kurt Akeley (Eds.). ACM, 103–112.
37. Ilya Kostrikov, Zhongshi Jiang, Daniele Panozzo, Denis Zorin, and Joan Bruna. 2018. Surface Networks. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018. IEEE Computer Society, 2540–2548.
38. Vladislav Kraevoy and Alla Sheffer. 2004. Cross-parameterization and compatible remeshing of 3D models. ACM Transactions on Graphics (TOG) 23, 3 (2004), 861–869.
39. Aaron W. F. Lee, Wim Sweldens, Peter Schröder, Lawrence C. Cowsar, and David P. Dobkin. 1998. MAPS: Multiresolution Adaptive Parameterization of Surfaces. In Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1998, Orlando, FL, USA, July 19-24, 1998, Steve Cunningham, Walt Bransford, and Michael F. Cohen (Eds.). ACM, 95–104.
40. Ruihui Li, Xianzhi Li, Chi-Wing Fu, Daniel Cohen-Or, and Pheng-Ann Heng. 2019. PU-GAN: A Point Cloud Upsampling Adversarial Network. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 – November 2, 2019. IEEE, 7202–7211.
41. Songrun Liu, Zachary Ferguson, Alec Jacobson, and Yotam I. Gingold. 2017. Seamless: seam erasure and seam-aware decoupling of shape from mesh resolution. ACM Trans. Graph. 36, 6 (2017), 216:1–216:15.
42. Yang Liu, Helmut Pottmann, Johannes Wallner, Yong-Liang Yang, and Wenping Wang. 2006. Geometric modeling with conical meshes and developable surfaces. In ACM transactions on graphics (TOG), Vol. 25. ACM, 681–689.
43. Charles Loop. 1987. Smooth subdivision surfaces based on triangles. Master’s thesis, University of Utah, Department of Mathematics (1987).
44. Haggai Maron, Meirav Galun, Noam Aigerman, Miri Trope, Nadav Dym, Ersin Yumer, Vladimir G. Kim, and Yaron Lipman. 2017. Convolutional neural networks on surfaces via seamless toric covers. ACM Trans. Graph. 36, 4 (2017), 71:1–71:10.
45. Jonathan Masci, Davide Boscaini, Michael M. Bronstein, and Pierre Vandergheynst. 2015. Geodesic Convolutional Neural Networks on Riemannian Manifolds. In 2015 IEEE International Conference on Computer Vision Workshop, ICCV Workshops 2015, Santiago, Chile, December 7-13, 2015. IEEE Computer Society, 832–840.
46. Patrick Mullen, Yiying Tong, Pierre Alliez, and Mathieu Desbrun. 2008. Spectral Conformal Parameterization. Comput. Graph. Forum 27, 5 (2008), 1487–1494.
47. Vinod Nair and Geoffrey E. Hinton. 2010. Rectified Linear Units Improve Restricted Boltzmann Machines. In Proceedings of the 27th International Conference on Machine Learning (ICML-10), June 21-24, 2010, Haifa, Israel, Johannes Fürnkranz and Thorsten Joachims (Eds.). Omnipress, 807–814.
48. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8-14 December 2019, Vancouver, BC, Canada, Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett (Eds.). 8024–8035.
49. Adrien Poulenard and Maks Ovsjanikov. 2018. Multi-directional geodesic neural networks via equivariant convolution. ACM Trans. Graph. 37, 6 (2018), 236:1–236:14.
50. Emil Praun, Wim Sweldens, and Peter Schröder. 2001. Consistent Mesh Parameterizations. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’01). Association for Computing Machinery, New York, NY, USA, 179–184.
51. Reinhold Preiner, Tamy Boubekeur, and Michael Wimmer. 2019. Gaussian-product subdivision surfaces. ACM Trans. Graph. 38, 4 (2019), 35:1–35:11.
52. Michael Rabinovich, Tim Hoffmann, and Olga Sorkine-Hornung. 2018. The shape space of discrete orthogonal geodesic nets. ACM Trans. Graph. 37, 6 (2018), 228:1–228:17.
53. Anurag Ranjan, Timo Bolkart, Soubhik Sanyal, and Michael J. Black. 2018. Generating 3D Faces Using Convolutional Mesh Autoencoders. In Computer Vision – ECCV 2018 – 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part III (Lecture Notes in Computer Science), Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (Eds.), Vol. 11207. Springer, 725–741.
54. Malcolm Sabin and Neil Dodgson. 2004. A Circle-Preserving Variant of the Four-Point Subdivision Scheme. Mathematical Methods for Curves and Surfaces: Tromsø 2004 (01 2004).
55. Scott Schaefer, E. Vouga, and Ron Goldman. 2008. Nonlinear subdivision through nonlinear averaging. Comput. Aided Geom. Des. 25, 3 (2008), 162–180.
56. John Schreiner, Arul Asirvatham, Emil Praun, and Hugues Hoppe. 2004. Inter-surface mapping. ACM Trans. Graph. 23, 3 (2004), 870–877.
57. Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhöfer. 2019. DeepVoxels: Learning Persistent 3D Feature Embeddings. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019. Computer Vision Foundation / IEEE, 2437–2446.
58. Olga Sorkine. 2005. Laplacian Mesh Processing. In Eurographics 2005 – State of the Art Reports, Dublin, Ireland, August 29 – September 2, 2005, Yiorgos Chrysanthou and Marcus A. Magnor (Eds.). Eurographics Association, 53–70.
59. Jos Stam. 1998. Exact Evaluation of Catmull-Clark Subdivision Surfaces at Arbitrary Parameter Values. In Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1998, Orlando, FL, USA, July 19-24, 1998, Steve Cunningham, Walt Bransford, and Michael F. Cohen (Eds.). ACM, 395–404.
60. Qingyang Tan, Lin Gao, Yu-Kun Lai, and Shihong Xia. 2018. Variational Autoencoders for Deforming 3D Mesh Models. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018. IEEE Computer Society, 5841–5850.
61. Chengcheng Tang, Xiang Sun, Alexandra Gomes, Johannes Wallner, and Helmut Pottmann. 2014. Form-Finding with Polyhedral Meshes Made Simple. ACM Trans. Graph. 33, 4, Article Article 70 (July 2014), 9 pages.
62. Maxim Tatarchenko, Stephan R. Richter, René Ranftl, Zhuwen Li, Vladlen Koltun, and Thomas Brox. 2019. What Do Single-View 3D Reconstruction Networks Learn?. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019. Computer Vision Foundation / IEEE, 3405–3414.
63. Robert F. Tobler, Stefan Maierhofer, and Alexander Wilkie. 2002a. Mesh-Based Parametrized L-Systems and Generalized Subdivision for Generating Complex Geometry. International Journal of Shape Modeling 8, 2 (2002), 173–191.
64. Robert F. Tobler, Stefan Maierhofer, and Alexander Wilkie. 2002b. A Multiresolution Mesh Generation Approach for Procedural Definition of Complex Geometry. In 2002 International Conference on Shape Modeling and Applications (SMI 2002), 17-22 May 2002, Banff, Alberta, Canada. IEEE Computer Society, 35–42.
65. Oliver Van Kaick, Hao Zhang, Ghassan Hamarneh, and Daniel Cohen-Or. 2011. A survey on shape correspondence. In Computer Graphics Forum, Vol. 30. Wiley Online Library, 1681–1707.
66. Amir Vaxman, Christian Müller, and Ofir Weber. 2018. Canonical Möbius subdivision. ACM Trans. Graph. 37, 6 (2018), 227:1–227:15.
67. Luiz Velho, Ken Perlin, Henning Biermann, and Lexing Ying. 2002. Algorithmic shape modeling with subdivision surfaces. Comput. Graph. 26, 6 (2002), 865–875.
68. Matthias Vestner, Roee Litman, Emanuele Rodolà, Alex Bronstein, and Daniel Cremers. 2017. Product manifold filter: Non-rigid shape correspondence via kernel density estimation in the product space. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3327–3336.
69. Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. 2018. Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images. In Computer Vision – ECCV 2018 – 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XI (Lecture Notes in Computer Science), Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (Eds.), Vol. 11215. Springer, 55–71.
70. Yu Wang, Vladimir G. Kim, Michael Bronstein, and Justin Solomon. 2019a. Learning Geometric Operators on Meshes. ICLR Workshop on Representation Learning on Graphs and Manifolds (2019).
71. Yifan Wang, Shihao Wu, Hui Huang, Daniel Cohen-Or, and Olga Sorkine-Hornung. 2019b. Patch-Based Progressive 3D Point Set Upsampling. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019. Computer Vision Foundation / IEEE, 5958–5967.
72. Chao Wen, Yinda Zhang, Zhuwen Li, and Yanwei Fu. 2019. Pixel2Mesh++: Multi-View 3D Mesh Generation via Deformation. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 – November 2, 2019. IEEE, 1042–1051.
73. Wang Yifan, Noam Aigerman, Vladimir G. Kim, Siddhartha Chaudhuri, and Olga Sorkine-Hornung. 2020. Neural Cages for Detail-Preserving 3D Deformations. In CVPR.
74. Lequan Yu, Xianzhi Li, Chi-Wing Fu, Daniel Cohen-Or, and Pheng-Ann Heng. 2018. PU-Net: Point Cloud Upsampling Network. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018. IEEE Computer Society, 2790–2799.
75. Qingnan Zhou and Alec Jacobson. 2016. Thingi10K: A Dataset of 10,000 3D-Printing Models. arXiv preprint arXiv:1605.04797 (2016). https://ten-thousand-models.appspot.com
76. Denis Zorin. 2007. Subdivision on arbitrary meshes: algorithms and theory. In Mathematics and Computation in Imaging Science and Information Processing. World Scientific, 1–46.
77. Denis Zorin, Peter Schröder, T De Rose, L Kobbelt, A Levin, and W Sweldens. 2000. Subdivision for modeling and animation. SIGGRAPH Course Notes (2000).
78. Denis Zorin, Peter Schröder, and Wim Sweldens. 1996. Interpolating Subdivision for Meshes with Arbitrary Topology. In Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’96). Association for Computing Machinery, New York, NY, USA, 189–192.
