“Convolutional neural networks on surfaces via seamless toric covers”

  • ©Haggai Maron, Meirav Galun, Noam Aigerman, Miri Trope, Nadav Dym, Ersin Yumer, Vladimir G. Kim, and Yaron Lipman




    Convolutional neural networks on surfaces via seamless toric covers



    The recent success of convolutional neural networks (CNNs) for image processing tasks is inspiring research efforts attempting to achieve similar success for geometric tasks. One of the main challenges in applying CNNs to surfaces is defining a natural convolution operator on surfaces.In this paper we present a method for applying deep learning to sphere-type shapes using a global seamless parameterization to a planar flat-torus, for which the convolution operator is well defined. As a result, the standard deep learning framework can be readily applied for learning semantic, high-level properties of the shape. An indication of our success in bridging the gap between images and surfaces is the fact that our algorithm succeeds in learning semantic information from an input of raw low-dimensional feature vectors.We demonstrate the usefulness of our approach by presenting two applications: human body segmentation, and automatic landmark detection on anatomical surfaces. We show that our algorithm compares favorably with competing geometric deep-learning algorithms for segmentation tasks, and is able to produce meaningful correspondences on anatomical surfaces where hand-crafted features are bound to fail.


    1. 2016. Adobe Fuse 3D Characters. https://www.mixamo.com. (2016). Accessed: 2016-10-15.Google Scholar
    2. 2016. Yobi3d – free 3d model search engine. https://www.yobi3d.com. (2016). Accessed: 2016-10-15.Google Scholar
    3. Noam Aigerman and Yaron Lipman. 2015. Orbifold tutte embeddings. ACM Trans. Graph 34, 6 (2015), 190.Google ScholarDigital Library
    4. Noam Aigerman and Yaron Lipman. 2016. Hyperbolic orbifold tutte embeddings. ACM Transactions on Graphics (TOG) 35, 6 (2016), 217.Google ScholarDigital Library
    5. Dragomir Anguelov, Praveen Srinivasan, Daphne Koller, Sebastian Thrun, Jim Rodgers, and James Davis. 2005. SCAPE: shape completion and animation of people. In ACM Transactions on Graphics (TOG), Vol. 24. ACM, 408–416. Google ScholarDigital Library
    6. Mathieu Aubry, Ulrich Schlickewei, and Daniel Cremers. 2011. The wave kernel signature: A quantum mechanical approach to shape analysis.. In ICCV Workshops. IEEE, 1626–1633. http://dblp.uni-trier.de/db/conf/iccvw/iccvw2011.html#AubrySC11 Google ScholarCross Ref
    7. Federica Bogo, Javier Romero, Matthew Loper, and Michael J. Black. 2014. FAUST: Dataset and evaluation for 3D mesh registration. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). IEEE, Piscataway, NJ, USA. Google ScholarDigital Library
    8. Davide Boscaini, Jonathan Masci, Simone Melzi, Michael M Bronstein, Umberto Castellani, and Pierre Vandergheynst. 2015. Learning class-specific descriptors for deformable shapes using localized spectral convolutional networks. In Computer Graphics Forum, Vol. 34. Wiley Online Library, 13–23.Google Scholar
    9. Davide Boscaini, Jonathan Masci, Emanuele Rodolà, and Michael M. Bronstein. 2016. Learning shape correspondence with anisotropic convolutional neural networks. In NIPS.Google Scholar
    10. Doug M Boyer, Yaron Lipman, Elizabeth St Clair, Jesus Puente, Biren A Patel, Thomas Funkhouser, Jukka Jernvall, and Ingrid Daubechies. 2011. Algorithms to automatically quantify the geometric similarity of anatomical surfaces. Proceedings of the National Academy of Sciences 108, 45 (2011), 18221–18226.Google ScholarCross Ref
    11. Leo Breiman. 2001. Random forests. Machine learning 45, 1 (2001), 5–32. Google ScholarDigital Library
    12. Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. 2013. Spectral networks and locally connected networks on graphs. arXiv preprint arXiv:1312.6203 (2013).Google Scholar
    13. Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. 2014. Semantic image segmentation with deep convolutional nets and fully connected crfs. arXiv preprint arXiv:1412.7062 (2014).Google Scholar
    14. E Brian Davies. 2007. Linear operators and their spectra. Vol. 106. Cambridge University Press. Google ScholarCross Ref
    15. Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. 2016. Convolutional neural networks on graphs with fast localized spectral filtering. In Advances in Neural Information Processing Systems. 3837–3845.Google Scholar
    16. Daniela Giorgi, Silvia Biasotti, and Laura Paraboschi. 2007. Shape retrieval contest 2007: Watertight models track. SHREC competition 8 (2007).Google Scholar
    17. Xianfeng Gu, Steven Gortler, and Hugues Hoppe. 2002. Geometry Images. In SIGGRAPH. Google ScholarDigital Library
    18. Kan Guo, Dongqing Zou, and Xiaowu Chen. 2015. 3D Mesh Labeling via Deep Convolutional Neural Networks. ACM Trans. Graph. 35, 1 (2015). Google ScholarDigital Library
    19. Mikael Henaff, Joan Bruna, and Yann LeCun. 2015. Deep convolutional networks on graph-structured data. arXiv preprint arXiv:1506.05163 (2015).Google Scholar
    20. Tao Ju. 2004. Robust repair of polygonal models. ACM Transactions on Graphics (TOG) 23, 3 (2004), 888–895. Google ScholarDigital Library
    21. Felix Kälberer, Matthias Nieser, and Konrad Polthier. 2007. QuadCover-Surface Parameterization using Branched Coverings. In Computer Graphics Forum, Vol. 26. Wiley Online Library, 375–384. Google ScholarCross Ref
    22. Evangelos Kalogerakis, Melinos Averkiou, Subhransu Maji, and Siddhartha Chaudhuri. 2016. 3D Shape Segmentation with Projective Convolutional Networks. arXiv preprint arXiv:1612.02808 (2016).Google Scholar
    23. Evangelos Kalogerakis, Aaron Hertzmann, and Karan Singh. 2010. Learning 3D mesh segmentation and labeling. ACM Transactions on Graphics (TOG) 29, 4 (2010), 102.Google ScholarDigital Library
    24. Vladimir G Kim, Yaron Lipman, and Thomas Funkhouser. 2011. Blended intrinsic maps. In ACM Transactions on Graphics (TOG), Vol. 30. ACM, 79.Google ScholarDigital Library
    25. Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 (2016).Google Scholar
    26. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems. 1097–1105.Google Scholar
    27. Dmitry Laptev, Nikolay Savinov, Joachim M Buhmann, and Marc Pollefeys. 2016. TIPOOLING: transformation-invariant pooling for feature learning in Convolutional Neural Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 289–297.Google Scholar
    28. Roee Litman and Alexander M Bronstein. 2014. Learning spectral descriptors for deformable shape correspondence. IEEE transactions on pattern analysis and machine intelligence 36, 1 (2014), 171–180. Google ScholarDigital Library
    29. Jonathan Long, Evan Shelhamer, and Trevor Darrell. 2015. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3431–3440. Google ScholarCross Ref
    30. Wolfgang Maier. 1984. Tooth morphology and dietary specialization. In Food acquisition and processing in primates. Springer, 303–330. Google ScholarCross Ref
    31. Jonathan Masci, Davide Boscaini, Michael Bronstein, and Pierre Vandergheynst. 2015. Geodesic convolutional neural networks on riemannian manifolds. In Proceedings of the IEEE International Conference on Computer Vision Workshops. 37–45. Google ScholarDigital Library
    32. J Milnor. 1965. Topology from a differentiable viewpoint(University of Virginia Press, Charlottesville, VA). (1965).Google Scholar
    33. Emil Praun and Hugues Hoppe. 2003. Spherical parametrization and remeshing. In ACM Transactions on Graphics (TOG), Vol. 22. ACM, 340–349. Google ScholarDigital Library
    34. Charles R. Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. 2016a. PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation. arXiv preprint arXiv:1612.00593 (2016).Google Scholar
    35. Charles Ruizhongtai Qi, Hao Su, Matthias Nießner, Angela Dai, Mengyuan Yan, and Leonidas Guibas. 2016b. Volumetric and Multi-View CNNs for Object Classification on 3D Data. In Proc. Computer Vision and Pattern Recognition (CVPR), IEEE. Google ScholarCross Ref
    36. Emanuele Rodolà, Samuel Rota Bulo, Thomas Windheuser, Matthias Vestner, and Daniel Cremers. 2014. Dense non-rigid shape correspondence using random forests. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4177–4184. Google ScholarDigital Library
    37. Ayan Sinha, Jing Bai, and Karthik Ramani. 2016. Deep learning 3D shape surfaces using geometry images. In European Conference on Computer Vision. Springer, 223–240. Google ScholarCross Ref
    38. Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned-Miller. 2015. Multi-view convolutional neural networks for 3d shape recognition. In Proceedings of the IEEE International Conference on Computer Vision. 945–953. Google ScholarDigital Library
    39. Jian Sun, Maks Ovsjanikov, and Leonidas Guibas. 2009. A Concise and Provably Informative Multi-scale Signature Based on Heat Diffusion. In Proceedings of the Symposium on Geometry Processing (SGP ’09). Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, 1383–1392. http://dl.acm.org/citation.cfm?id=1735603.1735621 Google ScholarCross Ref
    40. Federico Tombari, Samuele Salti, and Luigi Di Stefano. 2010. Unique signatures of histograms for local surface description. In European Conference on Computer Vision. Springer, 356–369. Google ScholarCross Ref
    41. W. T. Tutte. 1963. How to draw a graph. Proc. London Math. Soc. 13, 3 (1963), 743–768. Google ScholarCross Ref
    42. Andrea Vedaldi and Karel Lenc. 2015. Matconvnet: Convolutional neural networks for matlab. In Proceedings of the 23rd ACM international conference on Multimedia. ACM, 689–692. Google ScholarDigital Library
    43. Daniel Vlasic, Ilya Baran, Wojciech Matusik, and Jovan Popović. 2008. Articulated mesh animation from multi-view silhouettes. In ACM Transactions on Graphics (TOG), Vol. 27. ACM, 97. Google ScholarDigital Library
    44. Lingyu Wei, Qixing Huang, Duygu Ceylan, Etienne Vouga, and Hao Li. 2016. Dense Human Body Correspondences Using Convolutional Networks. In Proc. CVPR. Google ScholarCross Ref
    45. Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 2015. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1912–1920.Google Scholar
    46. Xinchen Yan, Jimei Yang, Ersin Yumer, Yijie Guo, and Honglak Lee. 2016. Perspective transformer nets: Learning single-view 3d object reconstruction without 3d supervision. In Advances in Neural Information Processing Systems. 1696–1704.Google Scholar
    47. Li Yi, Hao Su, Xingwen Guo, and Leonidas Guibas. 2016. SyncSpecCNN: Synchronized Spectral CNN for 3D Shape Segmentation. arXiv preprint arXiv:1612.00606 (2016).Google Scholar
    48. Hong-Kai Zhao, Stanley Osher, Barry Merriman, and Myungjoo Kang. 2000. Implicit and nonparametric shape reconstruction from unorganized data using a variational level set method. Computer Vision and Image Understanding 80, 3 (2000), 295–314. Google ScholarDigital Library

ACM Digital Library Publication: