“A Neural Galerkin Solver for Accurate Surface Reconstruction” by Huang, Chen and Hu
Conference:
Type(s):
Title:
- A Neural Galerkin Solver for Accurate Surface Reconstruction
Session/Category Title: Reconstruction and Repair
Presenter(s)/Author(s):
Abstract:
To reconstruct meshes from the widely-available 3D point cloud data, implicit shape representation is among the primary choices as an intermediate form due to its superior representation power and robustness in topological optimizations. Although different parameterizations of the implicit fields have been explored to model the underlying geometry, there is no explicit mechanism to ensure the fitting tightness of the surface to the input. We present in response, NeuralGalerkin, a neural Galerkin-method-based solver designed for reconstructing highly-accurate surfaces from the input point clouds. NeuralGalerkin internally discretizes the target implicit field as a linear combination of a set of spatially-varying basis functions inferred by an adaptive sparse convolution neural network. It then solves differentiably for a variational problem that incorporates both positional and normal constraints from the data in closed form within a single forward pass, highly respecting the raw input points. The reconstructed surface extracted from the implicit interpolants is hence very accurate and incorporates useful inductive biases benefiting from the training data. Extensive evaluations on various datasets demonstrate our method’s promising reconstruction performance and scalability.
References:
1. Noam Aigerman, Kunal Gupta, Vladimir G. Kim, Siddhartha Chaudhuri, Jun Saito, and Thibault Groueix. 2022. Neural Jacobian Fields: Learning Intrinsic Mappings of Arbitrary Meshes. ACM Trans. Graph. 41, 4, Article 109 (2022), 17 pages.
2. Matan Atzmon and Yaron Lipman. 2020a. Sal: Sign agnostic learning of shapes from raw data. In Proc. IEEE Conf. CVPR. 2565–2574.
3. Matan Atzmon and Yaron Lipman. 2020b. SALD: Sign Agnostic Learning with Derivatives. In Int. Conf. Learning Representations.
4. Dejan Azinović, Ricardo Martin-Brualla, Dan B Goldman, Matthias Nießner, and Justus Thies. 2021. Neural RGB-D surface reconstruction. arXiv preprint arXiv:2104.04532 (2021).
5. Jens Behley, Martin Garbade, Andres Milioto, Jan Quenzel, Sven Behnke, Cyrill Stachniss, and Jurgen Gall. 2019. Semantickitti: A dataset for semantic scene understanding of lidar sequences. In Proc. IEEE Int. Conf. Computer Vision. 9297–9307.
6. Yizhak Ben-Shabat and Stephen Gould. 2020. Deepfit: 3d surface fitting via neural network weighted least squares. In European Conf. Computer Vision. Springer, 20–34.
7. Federica Bogo, Javier Romero, Gerard Pons-Moll, and Michael J Black. 2017. Dynamic FAUST: Registering human bodies in motion. In Proc. IEEE Conf. CVPR. 6233–6242.
8. Fatih Calakli and Gabriel Taubin. 2011. SSD: Smooth Signed Distance Surface Reconstruction. Comput. Graph. Forum 30, 7 (2011), 1993–2002.
9. Jonathan C Carr, Richard K Beatson, Jon B Cherrie, Tim J Mitchell, W Richard Fright, Bruce C McCallum, and Tim R Evans. 2001. Reconstruction and representation of 3D objects with radial basis functions. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques. ACM, 67–76.
10. Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. 2017. Matterport3D: Learning from RGB-D Data in Indoor Environments. Int. Conf. 3D Vision (2017), 667–676.
11. Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. 2015. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012 (2015).
12. Haoxiang Chen, Jiahui Huang, Tai-Jiang Mu, and Shi-Min Hu. 2022. CIRCLE: Convolutional Implicit Reconstruction and Completion for Large-scale Indoor Scene. In European Conf. Computer Vision. Springer, 1000–1010.
13. Zhang Chen, Yinda Zhang, Kyle Genova, Sean Fanello, Sofien Bouaziz, Christian Häne, Ruofei Du, Cem Keskin, Thomas Funkhouser, and Danhang Tang. 2021. Multiresolution deep implicit functions for 3d shape representation. In Proc. IEEE Int. Conf. Computer Vision. 13087–13096.
14. Julian Chibane, Thiemo Alldieck, and Gerard Pons-Moll. 2020a. Implicit functions in feature space for 3d shape reconstruction and completion. In Proc. IEEE Conf. CVPR. 6970–6981.
15. Julian Chibane, Gerard Pons-Moll, et al. 2020b. Neural unsigned distance fields for implicit function learning. NeurIPS 33 (2020), 21638–21652.
16. Christopher B Choy, Danfei Xu, JunYoung Gwak, Kevin Chen, and Silvio Savarese. 2016. 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction. In European Conf. Computer Vision. Springer, 628–644.
17. Brian Curless and Marc Levoy. 1996. A volumetric method for building complex models from range images. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques. 303–312.
18. Philipp Erler, Paul Guerrero, Stefan Ohrhallinger, Niloy J Mitra, and Michael Wimmer. 2020. Points2surf learning implicit surfaces from point clouds. In European Conf. Computer Vision. Springer, 108–124.
19. Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In Int. Conf. Machine Learning. PMLR, 1126–1135.
20. BG Galerkin. 1915. On electrical circuits for the approximate solution of the Laplace equation. Vestnik Inzh 19 (1915), 897–908.
21. Andreas Geiger, Philip Lenz, and Raquel Urtasun. 2012. Are we ready for autonomous driving? the kitti vision benchmark suite. In Proc. IEEE Conf. CVPR. IEEE, 3354–3361.
22. Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, and Thomas Funkhouser. 2020. Local deep implicit functions for 3d shape. In Proc. IEEE Conf. CVPR. 4857–4866.
23. Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, and Thomas A Funkhouser. 2019. Deep Structured Implicit Functions. arXiv preprint arXiv:1912.06126 (2019).
24. Benjamin Graham, Martin Engelcke, and Laurens Van Der Maaten. 2018. 3d semantic segmentation with submanifold sparse convolutional networks. In Proc. IEEE Conf. CVPR. 9224–9232.
25. Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. 2020. Implicit Geometric Regularization for Learning Shapes. In Int. Conf. Machine Learning. PMLR, 3789–3799.
26. Benoit Guillard, Federico Stella, and Pascal Fua. 2021. MeshUDF: Fast and Differentiable Meshing of Unsigned Distance Field Networks. arXiv preprint arXiv:2111.14549 (2021).
27. Rana Hanocka, Gal Metzer, Raja Giryes, and Daniel Cohen-Or. 2020. Point2Mesh: a self-prior for deformable meshes. ACM Trans. Graph. 39, 4 (2020), 126–1.
28. Shi-Min Hu, Dun Liang, Guo-Ye Yang, Guo-Wei Yang, and Wen-Yang Zhou. 2020. Jittor: a novel deep learning framework with meta-operators and unified graph execution. Science China Information Sciences 63, 12 (2020), 1–21.
29. Jiahui Huang, Shi-Sheng Huang, Haoxuan Song, and Shi-Min Hu. 2021. DI-Fusion: Online Implicit 3D Reconstruction with Deep Priors. In Proc. IEEE Conf. CVPR. 8932–8941.
30. Zhangjin Huang, Yuxin Wen, Zihao Wang, Jinjuan Ren, and Kui Jia. 2022. Surface Reconstruction from Point Clouds: A Survey and a Benchmark. arXiv preprint arXiv:2205.02413 (2022).
31. Chiyu Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang, Matthias Nießner, Thomas Funkhouser, et al. 2020. Local implicit grid representations for 3d scenes. In Proc. IEEE Conf. CVPR. 6001–6010.
32. Yiwei Jin, Diqiong Jiang, and Ming Cai. 2020. 3d reconstruction using deep learning: a survey. Communications in Information and Systems 20, 4 (2020), 389–413.
33. Michael Kazhdan and Hugues Hoppe. 2013. Screened poisson surface reconstruction. ACM Trans. Graph. 32, 3 (2013), 1–13.
34. Michael M. Kazhdan, Matthew Bolitho, and Hugues Hoppe. 2006. Poisson surface reconstruction. In Proceedings of the Fourth Eurographics Symposium on Geometry Processing, Cagliari, Sardinia, Italy, June 26–28, 2006 (ACM International Conference Proceeding Series, Vol. 256), Alla Sheffer and Konrad Polthier (Eds.). Eurographics Association, 61–70.
35. Jianwei Li, Wei Gao, Yihong Wu, Yangdong Liu, and Yanfei Shen. 2022. High-quality indoor scene 3D reconstruction with RGB-D cameras: A brief review. Computational Visual Media (2022), 1–25.
36. Yiyi Liao, Simon Donne, and Andreas Geiger. 2018. Deep marching cubes: Learning explicit surface representations. In Proc. IEEE Conf. CVPR. 2916–2925.
37. Yaron Lipman. 2021. Phase Transitions, Distance Functions, and Implicit Neural Representations. In Int. Conf. Machine Learning. PMLR, 6702–6712.
38. Shi-Lin Liu, Hao-Xiang Guo, Hao Pan, Peng-Shuai Wang, Xin Tong, and Yang Liu. 2021. Deep implicit moving least-squares functions for 3d reconstruction. In Proc. IEEE Conf. CVPR. 1788–1797.
39. William E Lorensen and Harvey E Cline. 1987. Marching cubes: A high resolution 3D surface construction algorithm. ACM Trans. Graph. 21, 4 (1987), 163–169.
40. Julien NP Martel, David B Lindell, Connor Z Lin, Eric R Chan, Marco Monteiro, and Gordon Wetzstein. 2021. Acorn: adaptive coordinate networks for neural scene representation. ACM Trans. Graph. 40, 4 (2021), 1–13.
41. Donald Meagher. 1982. Geometric modeling using octree encoding. Computer graphics and image processing 19, 2 (1982), 129–147.
42. Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. 2019. Occupancy networks: Learning 3d reconstruction in function space. In Proc. IEEE Conf. CVPR. 4460–4470.
43. Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. 2022. Instant Neural Graphics Primitives with a Multiresolution Hash Encoding. ACM Trans. Graph. 41, 4, Article 102 (July 2022), 15 pages.
44. Ken Museth. 2013. VDB: High-resolution sparse volumes with dynamic topology. ACM Trans. Graph. 32, 3 (2013), 1–22.
45. Yutaka Ohtake, Alexander G. Belyaev, Marc Alexa, Greg Turk, and Hans-Peter Seidel. 2003. Multi-level partition of unity implicits. ACM Trans. Graph. 22, 3 (2003), 463–470.
46. Rasmus Pagh and Flemming Friche Rodler. 2004. Cuckoo hashing. Journal of Algorithms 51, 2 (2004), 122–144.
47. Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. 2019. Deepsdf: Learning continuous signed distance functions for shape representation. In Proc. IEEE Conf. CVPR. 165–174.
48. Songyou Peng, Chiyu Jiang, Yiyi Liao, Michael Niemeyer, Marc Pollefeys, and Andreas Geiger. 2021. Shape as points: A differentiable poisson solver. NeurIPS 34 (2021), 13032–13044.
49. Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger. 2020. Convolutional occupancy networks. In European Conf. Computer Vision. Springer, 523–540.
50. Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. 2017. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proc. IEEE Conf. CVPR. 652–660.
51. Sashank J Reddi, Satyen Kale, and Sanjiv Kumar. 2019. On the convergence of adam and beyond. arXiv preprint arXiv:1904.09237 (2019).
52. Raj Shekhar, Elias Fayyad, Roni Yagel, and J Fredrick Cornhill. 1996. Octree-based decimation of marching cubes surfaces. In Proceedings of Seventh Annual IEEE Visualization’96. IEEE, 335–342.
53. Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. 2020. Implicit neural representations with periodic activation functions. NeurIPS 33 (2020), 7462–7473.
54. Haoxuan Song, Jiahui Huang, Yan-Pei Cao, and Tai-Jiang Mu. 2021. HDR-Net-Fusion: Real-time 3D dynamic scene reconstruction with a hierarchical deep reinforcement network. Computational Visual Media 7, 4 (2021), 419–435.
55. Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler. 2021. Neural geometric level of detail: Real-time rendering with implicit 3D shapes. In Proc. IEEE Conf. CVPR. 11358–11367.
56. Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. 2020. Fourier features let networks learn high frequency functions in low dimensional domains. NeurIPS 33 (2020), 7537–7547.
57. Haotian Tang, Zhijian Liu, Xiuyu Li, Yujun Lin, and Song Han. 2022. TorchSparse: Efficient Point Cloud Inference Engine. In Conference on Machine Learning and Systems (MLSys).
58. Jiapeng Tang, Jiabao Lei, Dan Xu, Feiying Ma, Kui Jia, and Lei Zhang. 2021b. SAConvONet: Sign-Agnostic Optimization of Convolutional Occupancy Networks. In Proc. IEEE Int. Conf. Computer Vision. 6504–6513.
59. Jia-Heng Tang, Weikai Chen, Bo Wang, Songrun Liu, Bo Yang, Lin Gao, et al. 2021a. OctField: Hierarchical Implicit Functions for 3D Modeling. NeurIPS 34 (2021).
60. Maxim Tatarchenko, Stephan R Richter, René Ranftl, Zhuwen Li, Vladlen Koltun, and Thomas Brox. 2019. What do single-view 3d reconstruction networks learn?. In Proc. IEEE Conf. CVPR. 3405–3414.
61. Benjamin Ummenhofer and Vladlen Koltun. 2021. Adaptive Surface Reconstruction with Multiscale Convolutional Kernels. In Proc. IEEE Int. Conf. Computer Vision. 5651–5660.
62. Vecteezy. 2022. Nature Vectors by Vecteezy. Retrieved May 1, 2022 from https://www.vecteezy.com/free-vector/nature
63. Ignacio Vizzo, Xieyuanli Chen, Nived Chebrolu, Jens Behley, and Cyrill Stachniss. 2021. Poisson surface reconstruction for LiDAR odometry and mapping. In IEEE Int. Conf. Robotics Automation. IEEE, 5624–5630.
64. Peng-Shuai Wang, Yang Liu, Yu-Xiao Guo, Chun-Yu Sun, and Xin Tong. 2017. O-cnn: Octree-based convolutional neural networks for 3d shape analysis. ACM Trans. Graph. 36, 4 (2017), 1–11.
65. Peng-ShuaiWang, Yang Liu, and Xin Tong. 2022. Dual Octree Graph Networks for Learning Adaptive Volumetric Shape Representations. arXiv preprint arXiv:2205.02825 (2022).
66. Peng-Shuai Wang, Chun-Yu Sun, Yang Liu, and Xin Tong. 2018. Adaptive O-CNN: A patch-based deep representation of 3D shapes. ACM Trans. Graph. 37, 6 (2018), 1–11.
67. Francis Williams, Zan Gojcic, Sameh Khamis, Denis Zorin, Joan Bruna, Sanja Fidler, and Or Litany. 2021a. Neural Fields as Learnable Kernels for 3D Reconstruction. arXiv preprint arXiv:2111.13674 (2021).
68. Francis Williams, Teseo Schneider, Claudio Silva, Denis Zorin, Joan Bruna, and Daniele Panozzo. 2019. Deep geometric prior for surface reconstruction. In Proc. IEEE Conf. CVPR. 10130–10139.
69. Francis Williams, Matthew Trager, Joan Bruna, and Denis Zorin. 2021b. Neural splines: Fitting 3d surfaces with infinitely-wide neural networks. In Proc. IEEE Conf. CVPR. 9949–9958.
70. Jianglong Ye, Yuntao Chen, Naiyan Wang, and Xiaolong Wang. 2022. GIFS: Neural Implicit Function for General Shape Representation. arXiv preprint arXiv:2204.07126 (2022).
71. Jiwen Zhang. 1996. C-curves: an extension of cubic curves. Computer Aided Geometric Design 13, 3 (1996), 199–217.
72. Qingnan Zhou and Alec Jacobson. 2016. Thingi10k: A dataset of 10,000 3d-printing models. arXiv preprint arXiv:1605.04797 (2016).
