“DeltaConv: anisotropic operators for geometric deep learning on point clouds” by Wiersma, Nasikun, Eisemann and Hildebrandt

Learning from 3D point-cloud data has rapidly gained momentum, motivated by the success of deep learning on images and the increased availability of 3D data. In this paper, we aim to construct anisotropic convolution layers that work directly on the surface derived from a point cloud. This is challenging because of the lack of a global coordinate system for tangential directions on surfaces. We introduce DeltaConv, a convolution layer that combines geometric operators from vector calculus to enable the construction of anisotropic filters on point clouds. Because these operators are defined on scalar- and vector-fields, we separate the network into a scalar- and a vector-stream, which are connected by the operators. The vector stream enables the network to explicitly represent, evaluate, and process directional information. Our convolutions are robust and simple to implement and match or improve on state-of-the-art approaches on several benchmarks, while also speeding up training and inference.

1. Matan Atzmon, Haggai Maron, Yaron Lipman, Atzmon Matan, Maron Haggai, and Lipman Yaron. 2018. Point convolutional neural networks by extension operators. ACM Trans. Graph. 37, 4 (2018), 71:1–71:12.Google ScholarDigital Library
2. Yizhak Ben-Shabat, Michael Lindenbaum, and Anath Fischer. 2018. 3DmFV: Three-Dimensional Point Cloud Classification in Real-Time Using Convolutional Neural Networks. IEEE Robotics and Automation Letters 3 (2018), 3145–3152.Google ScholarCross Ref
3. Davide Boscaini, Jonathan Masci, Emanuele Rodolà, and Michael Bronstein. 2016. Learning shape correspondence with anisotropic convolutional neural networks. NeurIPS (2016).Google Scholar
4. Alexandre Boulch. 2020. ConvPoint: Continuous convolutions for point cloud processing. Comput. Graph. Forum 88 (2020), 24–34.Google ScholarCross Ref
5. Christopher Brandt, Leonardo Scandolo, Elmar Eisemann, and Klaus Hildebrandt. 2017. Spectral Processing of Tangential Vector Fields. Computer Graphics Forum 36, 6 (2017), 338–353.Google ScholarDigital Library
6. Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. 2021. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.13478 (2021).Google Scholar
7. Michael M. Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. 2017. Geometric Deep Learning: Going beyond Euclidean data. IEEE Signal Process. Mag. 34 (2017), 18–42.Google ScholarCross Ref
8. Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. 2014. Spectral Networks and Locally Connected Networks on Graphs. ICLR (2014).Google Scholar
9. Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. 2015. ShapeNet: An Information-Rich 3D Model Repository. Technical Report arXiv:1512.03012.Google Scholar
10. Jintai Chen, Biwen Lei, Qingyu Song, Haochao Ying, Danny Z Chen, and Jian Wu. 2020. A Hierarchical Graph Network for 3D Object Detection on Point Clouds. CVPR (2020).Google Scholar
11. Christopher Choy, JunYoung Gwak, and Silvio Savarese. 2019. 4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks. CVPR (2019).Google Scholar
12. Taco Cohen, Mario Geiger, Jonas Koehler, and Max Welling. 2018. Spherical CNNs. ICLR (2018).Google Scholar
13. Taco Cohen, Maurice Weiler, Berkay Kicanaoglu, and Max Welling. 2019. Gauge Equivariant Convolutional Networks and the Icosahedral CNN. ICML (2019).Google Scholar
14. Keenan Crane, Fernando de Goes, Mathieu Desbrun, and Peter Schroder. 2013a. Digital Geometry Processing with Discrete Exterior Calculus. SIGGRAPH Asia 2013 Courses (2013), 7:1–7:126.Google Scholar
15. Keenan Crane, Clarisse Weischedel, and Max Wardetzky. 2013b. Geodesics in heat: A new approach to computing distance based on heat flow. ACM Trans. Graph. 32, 5 (2013), 152:1–152:11.Google ScholarDigital Library
16. Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. 2017. ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. CVPR (2017).Google Scholar
17. Fernando de Goes, Mathieu Desbrun, and Yiying Tong. 2016. Vector Field Processing on Triangle Meshes. SIGGRAPH Asia 2016 Courses (2016), 27:1–27:49.Google Scholar
18. Pim de Haan, Maurice Weiler, Taco Cohen, and Max Welling. 2021. Gauge Equivariant Mesh CNNs: Anisotropic convolutions on geometric graphs. ICLR (2021).Google Scholar
19. C. Deng, O. Litany, Y. Duan, A. Poulenard, A. Tagliasacchi, and L. Guibas. 2021. Vector Neurons: A General Framework for SO(3)-Equivariant Networks. In ICCV.Google Scholar
20. Miguel Dominguez, Rohan Dhamdhere, Atir Petkar, Saloni Jain, Shagan Sah, and Raymond Ptucha. 2018. General-Purpose Deep Point Cloud Feature Extractor. WACV (2018).Google Scholar
21. Moshe Eliasof and Eran Treister. 2020. DiffGCN: Graph Convolutional Networks via Differential Operators and Algebraic Multigrid Pooling. NeurIPS (2020).Google Scholar
22. Carlos Esteves, Christine Allen-Blanchette, Ameesh Makadia, and Kostas Daniilidis. 2017. Learning SO(3) Equivariant Representations with Spherical CNNs. ECCV (2017).Google Scholar
23. Yifan Feng, Haoxuan You, Zizhao Zhang, Rongrong Ji, and Yue Gao. 2019. Hypergraph neural networks. AAAI (2019).Google Scholar
24. Matthias Fey and Jan E. Lenssen. 2019. Fast Graph Representation Learning with PyTorch Geometric. ICLR Workshop on Representation Learning on Graphs and Manifolds (2019).Google Scholar
25. Matthias Fey, Jan Eric Lenssen, Frank Weichert, and Heinrich Müller. 2018. SplineCNN: Fast Geometric Deep Learning with Continuous B-Spline Kernels. CVPR (2018).Google Scholar
26. Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. 2020. SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks. NeurIPS (2020).Google Scholar
27. Jan E Gerken, Jimmy Aronsson, Oscar Carlsson, Hampus Linander, Fredrik Ohlsson, Christoffer Petersson, and Daniel Persson. 2021. Geometric deep learning and equivariant neural networks. arXiv preprint arXiv:2105.13926 (2021).Google Scholar
28. Ankit Goyal, Hei Law, Bowei Liu, Alejandro Newell, and Jia Deng. 2021. Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline. ICML (2021).Google Scholar
29. Benjamin Graham, Martin Engelcke, and Laurens van der Maaten. 2018. 3D Semantic Segmentation with Submanifold Sparse Convolutional Networks. CVPR (2018).Google Scholar
30. Yulan Guo, Hanyun Wang, Qingyong Hu, Hao Liu, Li Liu, and Mohammed Bennamoun. 2020. Deep Learning for 3D Point Clouds: A Survey. IEEE TPAMI 43 (2020), 4338–4364.Google ScholarDigital Library
31. Rana Hanocka, Amir Hertz, Noa Fish, Raja Giryes, Shachar Fleishman, and Daniel Cohen-Or. 2019. MeshCNN: A Network with an Edge. ACM Trans. Graph. 38, 4 (2019), 90:1–90:12.Google ScholarDigital Library
32. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. CVPR (2016).Google Scholar
33. Pedro Hermosilla, Tobias Ritschel, Pere-Pau Vázquez, Àlvar Vinacua, and Timo Ropinski. 2018. Monte Carlo Convolution for Learning on Non-Uniformly Sampled Point Clouds. ACM Trans. Graph. 37, 6 (2018), 235:1–235:12.Google ScholarDigital Library
34. Binh-Son Hua, Quang-Hieu Pham, Duc Thanh Nguyen, Minh-Khoi Tran, Lap-Fai Yu, and Sai-Kit Yeung. 2016. SceneNN: A Scene Meshes Dataset with aNNotations. 3DV (2016).Google Scholar
35. Binh-Son Hua, Minh-Khoi Tran, and Sai-Kit Yeung. 2018. Pointwise Convolutional Neural Networks. CVPR (2018).Google Scholar
36. Sergey Ioffe and Christian Szegedy. 2015. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. ICML (2015).Google ScholarDigital Library
37. Chiyu Max Jiang, Jingwei Huang, Karthik Kashinath, Prabhat, Philip Marcus, and Matthias Nießner. 2019. Spherical CNNs on Unstructured Grids. ICLR (2019).Google Scholar
38. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In ICLR.Google Scholar
39. Thomas N. Kipf and Max Welling. 2017. Semi-Supervised Classification with Graph Convolutional Networks. ICLR (2017).Google Scholar
40. Ilya Kostrikov, Zhongshi Jiang, Daniele Panozzo, Denis Zorin, and Joan Bruna. 2018. Surface Networks. CVPR (2018).Google Scholar
41. Alon Lahav and A. Tal. 2020. MeshWalker: deep mesh understanding by random walks. ACM Trans. Graph. 39, 6 (2020), 263:1–263:13.Google ScholarDigital Library
42. Shiyi Lan, Ruichi Yu, Gang Yu, and L Davis. 2019. Modeling Local Geometric Structure of 3D Point Clouds Using Geo-CNN. CVPR (2019).Google Scholar
43. Eric-Tuan Le, Iasonas Kokkinos, and Niloy J Mitra. 2020. Going Deeper With Lean Point Networks. CVPR (2020).Google Scholar
44. Huan Lei, Naveed Akhtar, and Ajmal Mian. 2019. Octree guided CNN with Spherical Kernels for 3D Point Clouds. CVPR (2019).Google Scholar
45. Yangyan Li, Bu Rui, Mingchao Sun, Wei Wu, Xinhan Di, Baoquan Chen, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. 2018. PointCNN: Convolution on x-transformed points. NeurIPS (2018).Google Scholar
46. Zhouhui et al. Lian. 2011. SHREC ’11 Track: Shape Retrieval on Non-rigid 3D Watertight Meshes. Eurographics Workshop on 3D Object Retrieval (01 2011), 79–88.Google Scholar
47. Jian Liang and Hongkai Zhao. 2013. Solving Partial Differential Equations on Point Clouds. SIAM J. Sci. Comput. 35 (2013), A1461–A1486.Google ScholarCross Ref
48. Liqiang Lin, Pengdi Huang, Chi-Wing Fu, Kai Xu, Hao Zhang, and Hui Huang. 2020. One Point is All You Need: Directional Attention Point for Feature Learning. arXiv preprint arXiv:2012.06257 (2020).Google Scholar
49. Hsueh-Ti Derek Liu, Alec Jacobson, and Keenan Crane. 2017. A Dirac Operator for Extrinsic Shape Analysis. Comput. Graph. Forum 36, 5 (2017), 139–149.Google ScholarDigital Library
50. Jinxian Liu, Bingbing Ni, Caiyuan Li, Jiancheng Yang, and Qi Tian. 2019c. Dynamic Points Agglomeration for Hierarchical Point Sets Learning. ICCV (2019).Google Scholar
51. Weiping Liu, Jia Sun, Wanyi Li, Ting Hu, and Peng Wang. 2019d. Deep Learning on Point Clouds and Its Application: A Survey. Sens 19 (2019), 4188.Google ScholarCross Ref
52. Yongcheng Liu, Bin Fan, Gaofeng Meng, Jiwen Lu, Shiming Xiang, and Chunhong Pan. 2019a. DensePoint: Learning densely contextual representation for efficient point cloud processing. ICCV (2019).Google Scholar
53. Yongcheng Liu, Bin Fan, Shiming Xiang, and Chunhong Pan. 2019b. Relation-Shape Convolutional Neural Network for Point Cloud Analysis. CVPR (2019).Google Scholar
54. Ze Liu, Han Hu, Yue Cao, Zheng Zhang, and Xin Tong. 2020. A Closer Look at Local Aggregation Operators in Point Cloud Analysis. ECCV (2020).Google Scholar
55. Ilya Loshchilov and Frank Hutter. 2017. SGDR: Stochastic Gradient Descent with Warm Restarts. ICLR (2017).Google Scholar
56. Tao Lu, Limin Wang, and Gangshan Wu. 2021. CGA-Net: Category Guided Aggregation for Point Cloud Semantic Segmentation. CVPR (2021).Google Scholar
57. Francesco Milano, Antonio Loquercio, Antoni Rosinol, Davide Scaramuzza, and Luca Carlone. 2020. Primal-Dual Mesh Convolutional Neural Networks. NeurIPS (2020).Google Scholar
58. Thomas W. Mitchel, Vladimir G. Kim, and Michael Kazhdan. 2021. Field Convolutions for Surface CNNs. ICCV (2021).Google Scholar
59. Andrew Nealen. 2004. An As-Short-as-possible introduction to the least squares, weighted least squares and moving least squares methods for scattered data approximation and interpolation. URL: http://www.nealen.com/projects/mls/asapmls.pdf 1 (2004).Google Scholar
60. B. O’Neill. 1983. Semi-Riemannian Geometry With Applications to Relativity. Elsevier Science.Google Scholar
61. Guanghua Pan, Jun Wang, Rendong Ying, and Peilin Liu. 2018. 3DTI-Net: Learn Inner Transform Invariant 3D Geometry Features using Dynamic GCN. arXiv preprint arXiv:1812.06254 (2018).Google Scholar
62. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. NeurIPS (2019).Google Scholar
63. P Perona and J Malik. 1990. Scale-space and edge detection using anisotropic diffusion. IEEE TPAMI 12, 7 (1990), 629–639.Google ScholarDigital Library
64. Adrien Poulenard and Maks Ovsjanikov. 2018. Multi-directional geodesic neural networks via equivariant convolution. ACM Trans. Graph. 37, 6 (2018), 236:1–236:14.Google ScholarDigital Library
65. Adrien Poulenard, Marie-Julie Rakotosaona, Yann Ponty, and Maks Ovsjanikov. 2019. Effective Rotation-invariant Point CNN with Spherical Harmonics kernels. 3DV (2019).Google Scholar
66. Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. 2017a. PointNet: Deep learning on point sets for 3D classification and segmentation. CVPR (2017).Google Scholar
67. Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. 2017b. PointNet++: Deep hierarchical feature learning on point sets in a metric space. NeurIPS (2017).Google Scholar
68. Shi Qiu, Saeed Anwar, and Nick Barnes. 2021a. Dense-Resolution Network for Point Cloud Classification and Segmentation. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) (2021).Google ScholarCross Ref
69. Shi Qiu, Saeed Anwar, and Nick Barnes. 2021b. Geometric Back-projection Network for Point Cloud Classification. arXiv preprint arXiv:1911.12885 (2021).Google Scholar
70. Nicholas Sharp, Souhaib Attaiki, Keenan Crane, and Maks Ovsjanikov. 2021. Diffusion-Net: Discretization Agnostic Learning on Surfaces. ACM Trans. Graph. 41, 3 (2021), 27:1–27:16.Google Scholar
71. Yiru Shen, Chen Feng, Yaoqing Yang, and Dong Tian. 2018. Mining point cloud local structures by kernel correlation and graph pooling. CVPR (2018).Google Scholar
72. Martin Simonovsky and Nikos Komodakis. 2017. Dynamic edge-conditioned filters in convolutional neural networks on graphs. CVPR (2017).Google Scholar
73. Dmitriy Smirnov and Justin Solomon. 2021. HodgeNet: Learning Spectral Geometry on Triangle Meshes. ACM Trans. Graph. 40, 4 (2021), 166:1–166:11.Google ScholarDigital Library
74. Xiao Sun, Zhouhui Lian, and Jianguo Xiao. 2019. SRINet: Learning Strictly Rotation-Invariant Representations for Point Cloud Classification and Segmentation. ACM International Conference on Multimedia (2019), 980–988.Google ScholarDigital Library
75. Gusi Te, Wei Hu, Amin Zheng, and Zongming Guo. 2018. RGCNN: Regularized graph CNN for point cloud segmentation. ACM International Conference on Multimedia (2018), 746–754.Google ScholarDigital Library
76. Hugues Thomas, Charles R Qi, Jean-Emmanuel Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J Guibas. 2019. KPConv: Flexible and Deformable Convolution for Point Clouds. ICCV (2019).Google Scholar
77. Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. 2018. Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds. arXiv preprint arXiv:1802.08219 (2018).Google Scholar
78. Mikaela Angelina Uy, Quang-Hieu Pham, Binh-Son Hua, Duc Thanh Nguyen, and Sai-Kit Yeung. 2019. Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data. ICCV (2019).Google Scholar
79. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. NeurIPS (2017).Google Scholar
80. Chu Wang, Babak Samari, and Kaleem Siddiqi. 2018. Local spectral graph convolution for point set feature learning. ECCV (2018).Google Scholar
81. Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E. Sarma, Michael M. Bronstein, and Justin M. Solomon. 2019. Dynamic graph CNN for learning on point clouds. ACM Trans. Graph. 38, 5 (2019), 146:1–146:12.Google ScholarDigital Library
82. Max Wardetzky. 2006. Discrete Differential Operators on Polyhedral Surfaces – Convergence and Approximation. Ph.D. Dissertation. Freie Universiät Berlin.Google Scholar
83. Joachim Weickert. 1998. Anisotropic diffusion in image processing. Vol. 1.Google Scholar
84. Maurice Weiler, Patrick Forré, Erik Verlinde, and Max Welling. 2021. Coordinate Independent Convolutional Networks – Isometry and Gauge Equivariant Convolutions on Riemannian Manifolds. arXiv preprint arXiv:2106.06020 (2021).Google Scholar
85. Ruben Wiersma, Elmar Eisemann, and Klaus Hildebrandt. 2020. CNNs on surfaces using rotation-equivariant features. ACM Trans. Graph. 4 (2020), 92:1–92:12.Google Scholar
86. Wenxuan Wu, Zhongang Qi, and Li Fuxin. 2019. PointConv: Deep Convolutional Networks on 3D Point Clouds. CVPR (2019).Google Scholar
87. Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 2015. 3D ShapeNets: A deep representation for volumetric shapes. CVPR (2015).Google Scholar
88. Tiange Xiang, Chaoyi Zhang, Yang Song, Jianhui Yu, and Weidong Cai. 2021. Walk in the Cloud: Learning Curves for Point Clouds Shape Analysis. ICCV (2021).Google Scholar
89. Mutian Xu, Runyu Ding, Hengshuang Zhao, and Xiaojuan Qi. 2021a. PAConv: Position Adaptive Convolution With Dynamic Kernel Assembling on Point Clouds. CVPR (2021).Google Scholar
90. Mutian Xu, Junhao Zhang, Zhipeng Zhou, Mingye Xu, Xiaojuan Qi, and Yu Qiao. 2021b. Learning Geometry-Disentangled Representation for Complementary Understanding of 3D Object Point Cloud. AAAI (2021).Google Scholar
91. Yifan Xu, Tianqi Fan, Mingye Xu, Long Zeng, and Yu Qiao. 2018. SpiderCNN: Deep Learning on Point Sets with Parameterized Convolutional Filters. ECCV (2018).Google Scholar
92. Jiancheng Yang, Qiang Zhang, B Ni, L Li, J Liu, Mengdie Zhou, and Q Tian. 2019. Modeling Point Clouds With Self-Attention and Gumbel Subset Sampling. CVPR (2019).Google Scholar
93. Li Yi, Vladimir G. Kim, Duygu Ceylan, I-Chao Shen, Mengyan Yan, Hao Su, Cewu Lu, Qixing Huang, Alla Sheffer, and Leonidas Guibas. 2016. A Scalable Active Framework for Region Annotation in 3D Shape Collections. ACM Trans. Graph. 35, 6 (2016), 210:1–210:12.Google ScholarDigital Library
94. Cheng Zhang, Haocheng Wan, Shengqiang Liu, Xinyi Shen, and Zizhao Wu. 2021. PVT: Point-Voxel Transformer for 3D Deep Learning. arXiv preprint arXiv:2108.06076 (2021).Google Scholar
95. Kuangen Zhang, Ming Hao, Jing Wang, Clarence W de Silva, and Chenglong Fu. 2019. Linked Dynamic Graph CNN: Learning on Point Cloud via Linking Hierarchical Features. arXiv preprint arXiv:1904.10014 (2019).Google Scholar
96. Yingxue Zhang and Michael Rabbat. 2018. A Graph-CNN for 3D point cloud classification. ICASSP (2018), 6279–6283.Google Scholar
97. Hengshuang Zhao, Li Jiang, Chi-Wing Fu, and Jiaya Jia. 2019. PointWeb: Enhancing Local Neighborhood Features for Point Cloud Processing. CVPR (2019).Google Scholar
98. Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip H.S. Torr, and Vladlen Koltun. 2021. Point Transformer. ICCV (2021).Google Scholar