“Point convolutional neural networks by extension operators” by Maron, Atzmon and Lipman

  • ©Haggai Maron, Matan Atzmon, and Yaron Lipman



Entry Number: 71


    Point convolutional neural networks by extension operators

Session/Category Title: Image & Shape Analysis With CNNs




    This paper presents Point Convolutional Neural Networks (PCNN): a novel framework for applying convolutional neural networks to point clouds. The framework consists of two operators: extension and restriction, mapping point cloud functions to volumetric functions and vise-versa. A point cloud convolution is defined by pull-back of the Euclidean volumetric convolution via an extension-restriction mechanism.The point cloud convolution is computationally efficient, invariant to the order of points in the point cloud, robust to different samplings and varying densities, and translation invariant, that is the same convolution kernel is used at all points. PCNN generalizes image CNNs and allows readily adapting their architectures to the point cloud setting.Evaluation of PCNN on three central point cloud learning benchmarks convincingly outperform competing point cloud learning methods, and the vast majority of methods working with more informative shape representations such as surfaces and/or normals.


    1. Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2015. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. (2015). https://www.tensorflow.org/ Software available from tensorflow.org.Google Scholar
    2. Mikhail Belkin and Partha Niyogi. 2005. Towards a theoretical foundation for Laplacian-based manifold methods.. In COLT, Vol. 3559. Springer, 486–500. Google ScholarDigital Library
    3. Jon Louis Bentley. 1975. Multidimensional binary search trees used for associative searching. Commun. ACM 18, 9 (1975), 509–517. Google ScholarDigital Library
    4. Matthew Berger, Andrea Tagliasacchi, Lee M Seversky, Pierre Alliez, Gael Guennebaud, Joshua A Levine, Andrei Sharf, and Claudio T Silva. 2017. A survey of surface reconstruction from point clouds. In Computer Graphics Forum, Vol. 36. Wiley Online Library, 301–329. Google ScholarDigital Library
    5. Davide Boscaini, Jonathan Masci, Emanuele Rodolà, and Michael M. Bronstein. 2016. Learning shape correspondence with anisotropic convolutional neural networks. In NIPS. Google ScholarDigital Library
    6. Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. 2016. Generative and discriminative voxel modeling with convolutional neural networks. arXiv preprint arXiv:1608.04236 (2016).Google Scholar
    7. Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. 2017. Geometric deep learning: going beyond euclidean data. IEEE Signal Processing Magazine 34, 4 (2017), 18–42.Google ScholarCross Ref
    8. David S Broomhead and David Lowe. 1988. Radial basis functions, multi-variable functional interpolation and adaptive networks. Technical Report. Royal Signals and Radar Establishment Malvern (United Kingdom).Google Scholar
    9. Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. 2013. Spectral networks and locally connected networks on graphs. arXiv preprint arXiv:1312.6203 (2013).Google Scholar
    10. Yu D Burago and Viktor A Zalgaller. 2013. Geometry III: theory of surfaces. Vol. 48. Springer Science & Business Media.Google Scholar
    11. Jonathan C Carr, Richard K Beatson, Jon B Cherrie, Tim J Mitchell, W Richard Fright, Bruce C McCallum, and Tim R Evans. 2001. Reconstruction and representation of 3D objects with radial basis functions. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques. ACM, 67–76. Google ScholarDigital Library
    12. Özgün Çiçek, Ahmed Abdulkadir, Soeren S Lienkamp, Thomas Brox, and Olaf Ronneberger. 2016. 3D U-Net: learning dense volumetric segmentation from sparse annotation. In International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 424–432.Google Scholar
    13. Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. 2017. Deformable Convolutional Networks. arXiv preprint arXiv:1703.06211 (2017).Google Scholar
    14. Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. 2016. Convolutional neural networks on graphs with fast localized spectral filtering. In Advances in Neural Information Processing Systems. 3837–3845. Google ScholarDigital Library
    15. Manfredo P Do Carmo, Gerd Fischer, Ulrich Pinkall, and Helmut Reckziegel. 2017. Differential Geometry. In Mathematical Models. Springer, 155–180.Google Scholar
    16. Paul Guerrero, Yanir Kleiman, Maks Ovsjanikov, and Niloy J Mitra. 2017. PCP-NET: Learning Local Shape Properties from Raw Point Clouds. arXiv preprint arXiv:1710.04954 (2017).Google Scholar
    17. Vishakh Hegde and Reza Zadeh. 2016. Fusionnet: 3d object classification using multiple data representations. arXiv preprint arXiv:1607.05695 (2016).Google Scholar
    18. Mikael Henaff, Joan Bruna, and Yann LeCun. 2015. Deep convolutional networks on graph-structured data. arXiv preprint arXiv:1506.05163 (2015).Google Scholar
    19. Evangelos Kalogerakis, Melinos Averkiou, Subhransu Maji, and Siddhartha Chaudhuri. 2016. 3D Shape Segmentation with Projective Convolutional Networks. arXiv preprint arXiv:1612.02808 (2016).Google Scholar
    20. Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. 2017. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196 (2017).Google Scholar
    21. Roman Klokov and Victor Lempitsky. 2017. Escape from Cells: Deep Kd-Networks for The Recognition of 3D Point Cloud Models. arXiv preprint arXiv:1704.01222 (2017).Google Scholar
    22. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems. 1097–1105. Google ScholarDigital Library
    23. Yangyan Li, Rui Bu, Mingchao Sun, and Baoquan Chen. 2018. PointCNN. arXiv preprint arXiv:1801.07791 (2018).Google Scholar
    24. Yangyan Li, Soeren Pirk, Hao Su, Charles R Qi, and Leonidas J Guibas. 2016. Fpnn: Field probing neural networks for 3d data. In Advances in Neural Information Processing Systems. 307–315. Google ScholarDigital Library
    25. Haggai Maron, Meirav Galun, Noam Aigerman, Miri Trope, Nadav Dym, Ersin Yumer, VLADIMIR G KIM, and Yaron Lipman. 2017. Convolutional Neural Networks on Surfaces via Seamless Toric Covers. SIGGRAPH. Google ScholarDigital Library
    26. Jonathan Masci, Davide Boscaini, Michael Bronstein, and Pierre Vandergheynst. 2015. Geodesic convolutional neural networks on riemannian manifolds. In Proceedings of the IEEE International Conference on Computer Vision Workshops. 37–45. Google ScholarDigital Library
    27. Daniel Maturana and Sebastian Scherer. 2015. Voxnet: A 3d convolutional neural network for real-time object recognition. In Intelligent Robots and Systems (IROS), 2015 IEEE/RSJ International Conference on. IEEE, 922–928.Google ScholarCross Ref
    28. Federico Monti, Davide Boscaini, Jonathan Masci, Emanuele Rodolà, Jan Svoboda, and Michael M Bronstein. 2016. Geometric deep learning on graphs and manifolds using mixture model CNNs. arXiv preprint arXiv:1611.08402 (2016).Google Scholar
    29. Elizbar A Nadaraya. 1964. On estimating regression. Theory of Probability & Its Applications 9, 1 (1964), 141–142.Google ScholarCross Ref
    30. Mark JL Orr. 1999. Recent advances in radial basis function networks. Institute for Adaptative and Neural Computation (1999).Google Scholar
    31. Mark JL Orr et al. 1996. Introduction to radial basis function networks. (1996).Google Scholar
    32. Jooyoung Park and Irwin W Sandberg. 1991. Universal approximation using radial-basis-function networks. Neural computation 3, 2 (1991), 246–257.Google Scholar
    33. Charles R. Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. 2016a. PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation. arXiv preprint arXiv:1612.00593 (2016).Google Scholar
    34. Charles Ruizhongtai Qi, Hao Su, Matthias Nießner, Angela Dai, Mengyuan Yan, and Leonidas Guibas. 2016b. Volumetric and Multi-View CNNs for Object Classification on 3D Data. In Proc. Computer Vision and Pattern Recognition (CVPR), IEEE.Google ScholarCross Ref
    35. Charles R Qi, Li Yi, Hao Su, and Leonidas J Guibas. 2017. PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space. arXiv preprint arXiv:1706.02413 (2017).Google Scholar
    36. Siamak Ravanbakhsh, Jeff Schneider, and Barnabas Poczos. 2016. Deep learning with sets and point clouds. arXiv preprint arXiv:1611.04500 (2016).Google Scholar
    37. Gernot Riegler, Ali Osman Ulusoys, and Andreas Geiger. 2016. Octnet: Learning deep 3d representations at high resolutions. arXiv preprint arXiv:1611.05009 (2016).Google Scholar
    38. Kristof Schütt, Pieter-Jan Kindermans, Huziel Enoc Sauceda Felix, Stefan Chmiela, Alexandre Tkatchenko, and Klaus-Robert Müller. 2017. MolecuLeNet: A continuous-filter convolutional neural network for modeling quantum interactions. In Advances in Neural Information Processing Systems. 992–1002.Google Scholar
    39. Martin Simonovsky and Nikos Komodakis. 2017. Dynamic Edge-Conditioned Filters in Convolutional Neural Networks on Graphs. arXiv preprint arXiv:1704.02901 (2017).Google Scholar
    40. Ayan Sinha, Jing Bai, and Karthik Ramani. 2016. Deep learning 3D shape surfaces using geometry images. In European Conference on Computer Vision. Springer, 223–240.Google ScholarCross Ref
    41. Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned-Miller. 2015. Multi-view convolutional neural networks for 3d shape recognition. In Proceedings of the IEEE International Conference on Computer Vision. 945–953. Google ScholarDigital Library
    42. Peng-Shuai Wang, Yang Liu, Yu-Xiao Guo, Chun-Yu Sun, and Xin Tong. 2017. O-cnn: Octree-based convolutional neural networks for 3d shape analysis. ACM Transactions on Graphics (TOG) 36, 4 (2017), 72. Google ScholarDigital Library
    43. Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. 2018. Dynamic Graph CNN for Learning on Point Clouds. arXiv preprint arXiv:1801.07829 (2018).Google Scholar
    44. Eric W Weisstein. 2000. Normal sum distribution. (2000).Google Scholar
    45. Holger Wendland. 2004. Scattered data approximation. Vol. 17. Cambridge university press.Google Scholar
    46. Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 2015. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1912–1920.Google Scholar
    47. Li Yi, Vladimir G Kim, Duygu Ceylan, I Shen, Mengyan Yan, Hao Su, ARCewu Lu, Qixing Huang, Alia Sheffer, Leonidas Guibas, et al. 2016a. A scalable active framework for region annotation in 3d shape collections. ACM Transactions on Graphics (TOG) 35, 6 (2016), 210. Google ScholarDigital Library
    48. Li Yi, Hao Su, Xingwen Guo, and Leonidas Guibas. 2016b. SyncSpecCNN: Synchronized Spectral CNN for 3D Shape Segmentation. arXiv preprint arXiv:1612.00606 (2016).Google Scholar
    49. Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan Salakhutdinov, and Alexander Smola. 2017. Deep Sets. arXiv preprint arXiv:1703.06114 (2017).Google Scholar

ACM Digital Library Publication: