“Unsupervised Learning of Robust Spectral Shape Matching” by Cao, Roetzer and Bernard
Conference:
Type:
Title:
- Unsupervised Learning of Robust Spectral Shape Matching
Session/Category Title: Marvelous Mappings
Presenter(s)/Author(s):
Moderator(s):
Abstract:
We propose a novel learning-based approach for robust 3D shape matching. Our method builds upon deep functional maps and can be trained in a fully unsupervised manner. Previous deep functional map methods mainly focus on predicting optimised functional maps alone, and then rely on off-the-shelf post-processing to obtain accurate point-wise maps during inference. However, this two-stage procedure for obtaining point-wise maps often yields sub-optimal performance. In contrast, building upon recent insights about the relation between functional maps and point-wise maps, we propose a novel unsupervised loss to couple the functional maps and point-wise maps, and thereby directly obtain point-wise maps without any post-processing. Our approach obtains accurate correspondences not only for near-isometric shapes, but also for more challenging non-isometric shapes and partial shapes, as well as shapes with different discretisation or topological noise. Using a total of nine diverse datasets, we extensively evaluate the performance and demonstrate that our method substantially outperforms previous state-of-the-art methods, even compared to recent supervised methods. Our code is available at https://github.com/dongliangcao/Unsupervised-Learning-of-Robust-Spectral-Shape-Matching.
References:
1. Dragomir Anguelov, Praveen Srinivasan, Daphne Koller, Sebastian Thrun, Jim Rodgers, and James Davis. 2005. Scape: shape completion and animation of people. In ACM SIGGRAPH.
2. Souhaib Attaiki and Maks Ovsjanikov. 2022. NCP: Neural Correspondence Prior for Effective Unsupervised Shape Matching. In NeurIPS.
3. Souhaib Attaiki, Gautam Pai, and Maks Ovsjanikov. 2021. Dpfm: Deep partial functional maps. In International Conference on 3D Vision (3DV).
4. Mathieu Aubry, Ulrich Schlickewei, and Daniel Cremers. 2011. The wave kernel signature: A quantum mechanical approach to shape analysis. In ICCV.
5. Omri Azencot and Rongjie Lai. 2021. A Data-Driven Approach to Functional Map Construction and Bases Pursuit. In Computer Graphics Forum, Vol. 40. Wiley Online Library, 97–110.
6. Florian Bernard, Zeeshan Khan Suri, and Christian Theobalt. 2020. Mina: Convex mixed-integer programming for non-rigid shape alignment. In CVPR.
7. Florian Bernard, Johan Thunberg, Paul Swoboda, and Christian Theobalt. 2019. Hippi: Higher-order projected power iterations for scalable multi-matching. In ICCV.
8. Federica Bogo, Javier Romero, Matthew Loper, and Michael J Black. 2014. FAUST: Dataset and evaluation for 3D mesh registration. In CVPR.
9. Michael M Bronstein and Iasonas Kokkinos. 2010. Scale-invariant heat kernel signatures for non-rigid shape recognition. In CVPR.
10. Dongliang Cao and Florian Bernard. 2022. Unsupervised Deep Multi-shape Matching. In ECCV.
11. Aharon Cohen and Mirela Ben-Chen. 2020. Robust shape collection matching and correspondence from shape differences. In Computer Graphics Forum. Wiley Online Library.
12. Luca Cosmo, Emanuele Rodola, Michael M Bronstein, Andrea Torsello, Daniel Cremers, and Y Sahillioglu. 2016. SHREC’16: Partial matching of deformable shapes. Proc. 3DOR 2, 9 (2016), 12.
13. Huong Quynh Dinh, Anthony Yezzi, and Greg Turk. 2005. Texture transfer during shape transformation. ACM Transactions on Graphics (ToG) 24, 2 (2005), 289–310.
14. Nicolas Donati, Etienne Corman, Simone Melzi, and Maks Ovsjanikov. 2022b. Complex functional maps: A conformal link between tangent bundles. In Computer Graphics Forum. Wiley Online Library.
15. Nicolas Donati, Etienne Corman, and Maks Ovsjanikov. 2022a. Deep Orientation-Aware Functional Maps: Tackling Symmetry Issues in Shape Matching. In CVPR.
16. Nicolas Donati, Abhishek Sharma, and Maks Ovsjanikov. 2020. Deep geometric functional maps: Robust feature learning for shape correspondence. In CVPR.
17. Roberto M. Dyke, Yu-Kun Lai, Paul L. Rosin, Stefano Zappalà, Seana Dykes, Daoliang Guo, Kun Li, Riccardo Marin, Simone Melzi, and Jingyu Yang. 2020. SHREC’20: Shape correspondence with non-isometric deformations. Computers & Graphics 92 (2020), 28–43.
18. Bernhard Egger, William AP Smith, Ayush Tewari, Stefanie Wuhrer, Michael Zollhoefer, Thabo Beeler, Florian Bernard, Timo Bolkart, Adam Kortylewski, Sami Romdhani, et al. 2020. 3d morphable face models—past, present, and future. ACM Transactions on Graphics (ToG) 39, 5 (2020), 1–38.
19. Marvin Eisenberger, Zorah Lähner, and Daniel Cremers. 2019. Divergence-Free Shape Correspondence by Deformation. In Computer Graphics Forum. Wiley Online Library.
20. Marvin Eisenberger, Zorah Lahner, and Daniel Cremers. 2020a. Smooth shells: Multi-scale shape registration with functional maps. In CVPR.
21. Marvin Eisenberger, David Novotny, Gael Kerchenbaum, Patrick Labatut, Natalia Neverova, Daniel Cremers, and Andrea Vedaldi. 2021. Neuromorph: Unsupervised shape interpolation and correspondence in one go. In CVPR.
22. Marvin Eisenberger, Aysim Toker, Laura Leal-Taixé, and Daniel Cremers. 2020b. Deep shells: Unsupervised shape correspondence with optimal transport. NIPS (2020).
23. Davide Eynard, Emanuele Rodola, Klaus Glashoff, and Michael M Bronstein. 2016. Coupled functional maps. In 2016 Fourth International Conference on 3D Vision (3DV).
24. Danielle Ezuz and Mirela Ben-Chen. 2017. Deblurring and denoising of maps between shapes. In Computer Graphics Forum. Wiley Online Library.
25. Danielle Ezuz, Behrend Heeren, Omri Azencot, Martin Rumpf, and Mirela Ben-Chen. 2019a. Elastic correspondence between triangle meshes. In Computer Graphics Forum, Vol. 38. Wiley Online Library, 121–134.
26. Danielle Ezuz, Justin Solomon, and Mirela Ben-Chen. 2019b. Reversible harmonic maps between discrete surfaces. ACM Transactions on Graphics (ToG) 38, 2 (2019), 1–12.
27. Maolin Gao, Zorah Lahner, Johan Thunberg, Daniel Cremers, and Florian Bernard. 2021. Isometric multi-shape matching. In CVPR.
28. Thibault Groueix, Matthew Fisher, Vladimir G Kim, Bryan C Russell, and Mathieu Aubry. 2018. 3d-coded: 3d correspondences by deep deformation. In ECCV.
29. Oshri Halimi, Or Litany, Emanuele Rodola, Alex M Bronstein, and Ron Kimmel. 2019. Unsupervised learning of dense shape correspondence. In CVPR.
30. Benjamin Holzschuh, Zorah Lähner, and Daniel Cremers. 2020. Simulated annealing for 3d shape correspondence. In 2020 International Conference on 3D Vision (3DV).
31. Qixing Huang, Fan Wang, and Leonidas Guibas. 2014. Functional map networks for analyzing and exploring large shape collections. ACM Transactions on Graphics (ToG) 33, 4 (2014), 1–11.
32. Qi-Xing Huang, Bart Adams, Martin Wicke, and Leonidas J Guibas. 2008. Non-rigid registration under isometric deformations. In Computer Graphics Forum. Wiley Online Library.
33. Qi-Xing Huang and Leonidas Guibas. 2013. Consistent shape maps via semidefinite programming. In Computer Graphics Forum. Wiley Online Library.
34. Ruqi Huang, Jing Ren, Peter Wonka, and Maks Ovsjanikov. 2020. Consistent zoomout: Efficient spectral map synchronization. In Computer Graphics Forum. Wiley Online Library.
35. Eric Jang, Shixiang Gu, and Ben Poole. 2016. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144 (2016).
36. Vladimir G Kim, Yaron Lipman, and Thomas Funkhouser. 2011. Blended intrinsic maps. ACM Transactions on Graphics (ToG) 30, 4 (2011), 1–12.
37. Diederik P Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In ICLR.
38. Artiom Kovnatsky, Michael M Bronstein, Alexander M Bronstein, Klaus Glashof, and Ron Kimmel. 2013. Coupled quasi-harmonic bases. In Computer Graphics Forum. Wiley Online Library.
39. Zorah Lähner, Emanuele Rodola, Michael M Bronstein, Daniel Cremers, Oliver Burghard, Luca Cosmo, Andreas Dieckmann, Reinhard Klein, and Yusuf Sahillioglu. 2016. SHREC’16: Matching of deformable shapes with topological noise. Proc. 3DOR 2, 10.2312 (2016).
40. Lei Li, Nicolas Donati, and Maks Ovsjanikov. 2022. Learning Multi-resolution Functional Maps with Spectral Attention for Robust Shape Matching. NIPS (2022).
41. Qinsong Li, Shengjun Liu, Ling Hu, and Xinru Liu. 2020. Shape correspondence using anisotropic Chebyshev spectral CNNs. In CVPR.
42. Yang Li, Hikari Takehara, Takafumi Taketomi, Bo Zheng, and Matthias Nießner. 2021. 4dcomplete: Non-rigid motion estimation beyond the observable surface. In ICCV.
43. Or Litany, Tal Remez, Emanuele Rodola, Alex Bronstein, and Michael Bronstein. 2017a. Deep functional maps: Structured prediction for dense shape correspondence. In ICCV.
44. Or Litany, Emanuele Rodolà, Alexander M Bronstein, and Michael M Bronstein. 2017b. Fully spectral partial shape matching. In Computer Graphics Forum. Wiley Online Library.
45. Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. 2015. SMPL: A skinned multi-person linear model. ACM Transactions on Graphics (ToG) 34, 6 (2015), 1–16.
46. Robin Magnet, Jing Ren, Olga Sorkine-Hornung, and Maks Ovsjanikov. 2022. Smooth non-rigid shape matching via effective Dirichlet energy optimization. In International Conference on 3D Vision (3DV).
47. Riccardo Marin, Marie-Julie Rakotosaona, Simone Melzi, and Maks Ovsjanikov. 2020. Correspondence learning via linearly-invariant embedding. In NeurIPS.
48. Simone Melzi, Riccardo Marin, Emanuele Rodolà, Umberto Castellani, Jing Ren, Adrien Poulenard, Peter Wonka, and Maks Ovsjanikov. 2019a. Shrec 2019: Matching humans with different connectivity. In Eurographics Workshop on 3D Object Retrieval.
49. Simone Melzi, Jing Ren, Emanuele Rodolà, Abhishek Sharma, Peter Wonka, and Maks Ovsjanikov. 2019b. ZoomOut: spectral upsampling for efficient shape correspondence. ACM Transactions on Graphics (ToG) 38, 6 (2019), 1–14.
50. Maks Ovsjanikov, Mirela Ben-Chen, Justin Solomon, Adrian Butscher, and Leonidas Guibas. 2012. Functional maps: a flexible representation of maps between shapes. ACM Transactions on Graphics (ToG) 31, 4 (2012), 1–11.
51. Maks Ovsjanikov, Quentin Mérigot, Facundo Mémoli, and Leonidas Guibas. 2010. One point isometric matching with the heat kernel. In Computer Graphics Forum. Wiley Online Library.
52. Gautam Pai, Jing Ren, Simone Melzi, Peter Wonka, and Maks Ovsjanikov. 2021. Fast sinkhorn filters: Using matrix scaling for non-rigid shape correspondence with functional maps. In CVPR.
53. Ulrich Pinkall and Konrad Polthier. 1993. Computing discrete minimal surfaces and their conjugates. Experimental mathematics 2, 1 (1993), 15–36.
54. Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. 2017. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. NIPS (2017).
55. Jing Ren, Simone Melzi, Maks Ovsjanikov, and Peter Wonka. 2020. MapTree: recovering multiple solutions in the space of maps. ACM Transactions on Graphics (ToG) 39, 6 (2020), 1–17.
56. Jing Ren, Simone Melzi, Peter Wonka, and Maks Ovsjanikov. 2021. Discrete optimization for shape matching. In Computer Graphics Forum. Wiley Online Library.
57. Jing Ren, Mikhail Panine, Peter Wonka, and Maks Ovsjanikov. 2019. Structured regularization of functional map computations. In Computer Graphics Forum. Wiley Online Library.
58. Jing Ren, Adrien Poulenard, Peter Wonka, and Maks Ovsjanikov. 2018. Continuous and orientation-preserving correspondences via functional maps. ACM Transactions on Graphics (ToG) 37 (2018), 1–16.
59. Emanuele Rodolà, Luca Cosmo, Michael M Bronstein, Andrea Torsello, and Daniel Cremers. 2017. Partial functional correspondence. In Computer Graphics Forum. Wiley Online Library.
60. Emanuele Rodolà, Michael Moeller, and Daniel Cremers. 2015. Point-wise map recovery and refinement from functional correspondence. arXiv preprint arXiv:1506.05603 (2015).
61. Emanuele Rodola, Michael Moeller, and Daniel Cremers. 2017. Regularized pointwise map recovery from functional correspondence. In Computer Graphics Forum. Wiley Online Library.
62. Paul Roetzer, Paul Swoboda, Daniel Cremers, and Florian Bernard. 2022. A Scalable Combinatorial Solver for Elastic Geometrically Consistent 3D Shape Matching. In CVPR.
63. Jean-Michel Roufosse, Abhishek Sharma, and Maks Ovsjanikov. 2019. Unsupervised deep learning for structured shape matching. In ICCV.
64. Yusuf Sahillioğlu. 2020. Recent advances in shape correspondence. The Visual Computer 36, 8 (2020), 1705–1721.
65. Samuele Salti, Federico Tombari, and Luigi Di Stefano. 2014. SHOT: Unique signatures of histograms for surface and texture description. Computer Vision and Image Understanding 125 (2014), 251–264.
66. Abhishek Sharma and Maks Ovsjanikov. 2020. Weakly supervised deep functional maps for shape matching. NIPS (2020).
67. Nicholas Sharp, Souhaib Attaiki, Keenan Crane, and Maks Ovsjanikov. 2020. Diffusion-net: Discretization agnostic learning on surfaces. arXiv preprint arXiv:2012.00888 (2020).
68. Robert W Sumner and Jovan Popović. 2004. Deformation transfer for triangle meshes. ACM Transactions on Graphics (ToG) 23, 3 (2004), 399–405.
69. Gary KL Tam, Zhi-Quan Cheng, Yu-Kun Lai, Frank C Langbein, Yonghuai Liu, David Marshall, Ralph R Martin, Xian-Fang Sun, and Paul L Rosin. 2012. Registration of 3D point clouds and meshes: A survey from rigid to nonrigid. IEEE transactions on visualization and computer graphics 19, 7 (2012), 1199–1217.
70. Hugues Thomas, Charles R Qi, Jean-Emmanuel Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J Guibas. 2019. Kpconv: Flexible and deformable convolution for point clouds. In ICCV.
71. Giovanni Trappolini, Luca Cosmo, Luca Moschella, Riccardo Marin, Simone Melzi, and Emanuele Rodolà. 2021. Shape registration in the time of transformers. NIPS (2021).
72. Oliver Van Kaick, Hao Zhang, Ghassan Hamarneh, and Daniel Cohen-Or. 2011. A survey on shape correspondence. In Computer Graphics Forum. Wiley Online Library.
73. Gul Varol, Javier Romero, Xavier Martin, Naureen Mahmood, Michael J Black, Ivan Laptev, and Cordelia Schmid. 2017. Learning from synthetic humans. In CVPR.
74. Matthias Vestner, Roee Litman, Emanuele Rodola, Alex Bronstein, and Daniel Cremers. 2017. Product manifold filter: Non-rigid shape correspondence via kernel density estimation in the product space. In CVPR.
75. Fan Wang, Qixing Huang, and Leonidas J Guibas. 2013. Image co-segmentation via consistent functional maps. In ICCV.
76. Ruben Wiersma, Elmar Eisemann, and Klaus Hildebrandt. 2020. Cnns on surfaces using rotation-equivariant features. ACM Transactions on Graphics (ToG) 39, 4 (2020), 92–1.
77. Thomas Windheuser, Ulrich Schlickewei, Frank R Schmidt, and Daniel Cremers. 2011. Geometrically consistent elastic matching of 3d shapes: A linear programming solution. In ICCV.
78. Silvia Zuffi, Angjoo Kanazawa, David W Jacobs, and Michael J Black. 2017. 3D menagerie: Modeling the 3D shape and pose of animals. In CVPR.
