“SHRED: 3D Shape Region Decomposition with Learned Local Operations” by Jones, Habib and Ritchie
Conference:
Type(s):
Title:
- SHRED: 3D Shape Region Decomposition with Learned Local Operations
Session/Category Title: Distances and Matching
Presenter(s)/Author(s):
Abstract:
We present SHRED, a method for 3D SHape REgion Decomposition. SHRED takes a 3D point cloud as input and uses learned local operations to produce a segmentation that approximates fine-grained part instances. We endow SHRED with three decomposition operations: splitting regions, fixing the boundaries between regions, and merging regions together. Modules are trained independently and locally, allowing SHRED to generate high-quality segmentations for categories not seen during training. We train and evaluate SHRED with fine-grained segmentations from PartNet; using its merge-threshold hyperparameter, we show that SHRED produces segmentations that better respect ground-truth annotations compared with baseline methods, at any desired decomposition granularity. Finally, we demonstrate that SHRED is useful for downstream applications, out-performing all baselines on zero-shot fine-grained part instance segmentation and few-shot finegrained semantic segmentation when combined with methods that learn to label shape regions.
References:
1. Ben Abbatematteo, Stefanie Tellex, and George Konidaris. 2019. Learning to Generalize Kinematic Models to Novel Objects. In Proceedings of the Third Conference on Robot Learning.
2. Shmuel Asafi, Avi Goren, and Daniel Cohen-Or. 2013. Weak convex decomposition by lines-of-sight. In Computer graphics forum, Vol. 32. Wiley Online Library, 23–31.
3. Zhiqin Chen, Andrea Tagliasacchi, and Hao Zhang. 2019a. BSP-Net: Generating Compact Meshes via Binary Space Partitioning. arXiv:1911.06971 [cs.CV]
4. Zhiqin Chen, Kangxue Yin, Matthew Fisher, Siddhartha Chaudhuri, and Hao Zhang. 2019b. BAE-NET: Branched Autoencoder for Shape Co-Segmentation. Proceedings of International Conference on Computer Vision (ICCV) (2019).
5. Boyang Deng, Kyle Genova, Soroosh Yazdani, Sofien Bouaziz, Geoffrey Hinton, and Andrea Tagliasacchi. 2020. CvxNet: Learnable Convex Decomposition. (June 2020).
6. Matheus Gadelha, Aruni RoyChowdhury, Gopal Sharma, Evangelos Kalogerakis, Liangliang Cao, Erik Learned-Miller, Rui Wang, and Subhransu Maji. 2020. Label-Efficient Learning on Point Clouds using Approximate Convex Decompositions. In European Conference on Computer Vision (ECCV).
7. Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sarna, William T Freeman, and Thomas Funkhouser. 2019. Learning shape templates with structured implicit functions. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 7154–7164.
8. Songfang Han, Jiayuan Gu, Kaichun Mo, Li Yi, Siyu Hu, Xuejin Chen, and Hao Su. 2020. Compositionally Generalizable 3D Structure Prediction. (2020). arXiv:arXiv:2012.02493
9. Yu Hao and Yi Fang. 2021. Meta-Learning 3D Shape Segmentation Functions.
10. Hao Huang, Xiang Li, Lingjing Wang, and Yi Fang. 2021. 3D-MetaConNet: Meta-learning for 3D Shape Classification and Segmentation. In 2021 International Conference on 3D Vision (3DV). 982–991.
11. R. Kenny Jones, Aalia Habib, Rana Hanocka, and Daniel Ritchie. 2022. The Neurally-Guided Shape Parser: Grammar-based Labeling of 3D Shape Regions with Approximate Inference. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
12. Oliver Van Kaick, Noa Fish, Yanir Kleiman, Shmuel Asafi, and Daniel Cohen-Or. 2014. Shape segmentation by approximate convexity analysis. ACM Transactions on Graphics (TOG) 34, 1 (2014), 1–11.
13. Sagi Katz and Ayellet Tal. 2003. Hierarchical mesh decomposition using fuzzy clustering and cuts. ACM transactions on graphics (TOG) 22, 3 (2003), 954–961.
14. Yuki Kawana, Yusuke Mukuta, and Tatsuya Harada. 2020. Neural Star Domain as Primitive Representation. In NeurIPS 2020.
15. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. CoRR abs/1412.6980 (2014).
16. Loic Landrieu and Martin Simonovsky. 2018. Large-scale point cloud semantic segmentation with superpoint graphs. In Proceedings of the IEEE conference on computer vision and pattern recognition. 4558–4567.
17. Jyh-Ming Lien and Nancy M Amato. 2008. Approximate convex decomposition of polyhedra and its applications. Computer Aided Geometric Design 25, 7 (2008), 503–522.
18. Rong Liu and Hao Zhang. 2004. Segmentation of 3D meshes through spectral clustering. In 12th Pacific Conference on Computer Graphics and Applications, 2004. PG 2004. Proceedings. 298–305.
19. Zhengzhe Liu, Xiaojuan Qi, and Chi-Wing Fu. 2021. One Thing One Click: A Self-Training Approach for Weakly Supervised 3D Semantic Segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1726–1736.
20. Tiange Luo, Kaichun Mo, Zhiao Huang, Jiarui Xu, Siyu Hu, Liwei Wang, and Hao Su. 2020. Learning to Group: A Bottom-Up Framework for 3D Part Discovery in Unseen Categories. In International Conference on Learning Representations. https://openreview.net/forum?id=rkl8dlHYvB
21. Khaled Mamou. 2016. Volumetric Hierarchical Approximate Convex Decomposition. In Game Engine Gems 3, Eric Lengyel (Ed.). A K Peters, 141–158.
22. Kaichun Mo, Shilin Zhu, Angel X. Chang, Li Yi, Subarna Tripathi, Leonidas J. Guibas, and Hao Su. 2019. PartNet: A Large-Scale Benchmark for Fine-Grained and Hierarchical Part-Level 3D Object Understanding. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
23. Despoina Paschalidou, Angelos Katharopoulos, Andreas Geiger, and Sanja Fidler. 2021. Neural Parts: Learning Expressive 3D Shape Abstractions with Invertible Neural Networks. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR).
24. Despoina Paschalidou, Ali Osman Ulusoy, and Andreas Geiger. 2019. Superquadrics Revisited: Learning 3D Shape Parsing beyond Cuboids. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR).
25. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch. (2017).
26. Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. 2017a. Pointnet: Deep learning on point sets for 3D classification and segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 652–660.
27. Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. 2017b. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In Advances in neural information processing systems. 5099–5108.
28. Ariel Shamir. 2008. A survey on mesh segmentation techniques. In Computer graphics forum, Vol. 27. Wiley Online Library, 1539–1556.
29. Gopal Sharma, Bidya Dash, Matheus Gadelha, Aruni RoyChowdhury, Marios Loizou, Evangelos Kalogerakis, Liangliang Cao, Erik Learned-Miller, and Rui Wang and- Subhransu Maji. 2021. SurFit: Learning to Fit Surfaces Improves Few Shot Learning on Point Clouds.
30. Gopal Sharma, Evangelos Kalogerakis, and Subhransu Maji. 2019. Learning Point Embeddings from Shape Repositories for Few-Shot Segmentation. In 2019 International Conference on 3D Vision, 3DV 2019, Québec City, QC, Canada, September 16–19, 2019. IEEE, 67–75.
31. Chunyu Sun, Yiqi Yang, Haoxiang Guo, pengshuai Wang, Xin Tong, Yang Liu, and Shum Heung-Yeung. 2022. Semi-Supervised 3D Shape Segmentation with Multilevel Consistency and Part Substitution. Computational Visual Media (2022).
32. Chun-Yu Sun, Qian-Fang Zou, Xin Tong, and Yang Liu. 2019. Learning Adaptive Hierarchical Cuboid Abstractions of 3D Shape Collections. ACM Trans. Graph. 38, 6, Article 241 (Nov. 2019), 13 pages.
33. Shubham Tulsiani, Hao Su, Leonidas J. Guibas, Alexei A. Efros, and Jitendra Malik. 2017. Learning Shape Abstractions by Assembling Volumetric Primitives. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
34. Lingjing Wang, Xiang Li, and Yi Fang. 2020. Few-Shot Learning of Part-Specific Probability Space for 3D Shape Segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
35. Peng-Shuai Wang, Yu-Qi Yang, Qian-Fang Zou, Zhirong Wu, Yang Liu, and Xin Tong. 2021b. Unsupervised 3D Learning for Shape Analysis via Multiresolution Instance Discrimination. Proceedings of the AAAI Conference on Artificial Intelligence 35, 4 (May 2021), 2773–2781. https://ojs.aaai.org/index.php/AAAI/article/view/16382
36. Weiyue Wang, Ronald Yu, Qiangui Huang, and Ulrich Neumann. 2018. SGPN: Similarity Group Proposal Network for 3D Point Cloud Instance Segmentation. In CVPR.
37. Xiaogang Wang, Xun Sun, Xinyu Cao, Kai Xu, and Bin Zhou. 2021a. Learning FineGrained Segmentation of 3D Shapes Without Part Labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 10276–10285.
38. Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E. Sarma, Michael M. Bronstein, and Justin M. Solomon. 2019. Dynamic Graph CNN for Learning on Point Clouds. ACM Transactions on Graphics (TOG) (2019).
39. Yanzhen Wang, Kai Xu, Jun Li, Hao Zhang, Ariel Shamir, Ligang Liu, Zhiquan Cheng, and Yueshan Xiong. 2011. Symmetry hierarchy of man-made objects. In Computer graphics forum, Vol. 30. Wiley Online Library, 287–296.
40. Erik Wijmans. 2018. Pointnet++ Pytorch. https://github.com/erikwijmans/Pointnet2_PyTorch (2018).
41. Saining Xie, Jiatao Gu, Demi Guo, Charles R. Qi, Leonidas Guibas, and Or Litany. 2020. PointContrast: Unsupervised Pre-training for 3D Point Cloud Understanding. In Computer Vision – ECCV 2020, Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (Eds.). Springer International Publishing, Cham, 574–591.
42. Xun Xu and Gim Hee Lee. 2020. Weakly Supervised Semantic Point Cloud Segmentation: Towards 10x Fewer Labels. In CVPR.
43. Kaizhi Yang and Xuejin Chen. 2021. Unsupervised Learning for Cuboid Shape Abstraction via Joint Segmentation from Point Clouds. ACM Trans. Graph. 40, 4, Article 152 (jul 2021), 11 pages.
44. Li Yi, Leonidas Guibas, Aaron Hertzmann, Vladimir G. Kim, Hao Su, and Ersin Yumer. 2017. Learning Hierarchical Shape Segmentation and Labeling from Online Repositories. SIGGRAPH (2017).
45. Li Yi, Wang Zhao, He Wang, Minhyuk Sung, and Leonidas J. Guibas. 2019. GSPN: Generative Shape Proposal Network for 3D Instance Segmentation in Point Cloud. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 3942–3951.
46. Fenggen Yu, Kun Liu, Yan Zhang, Chenyang Zhu, and Kai Xu. 2019. PartNet: A Recursive Part Decomposition Network for Fine-grained and Hierarchical Shape Segmentation. In CVPR. to appear.
47. Chenyang Zhu, Kai Xu, Siddhartha Chaudhuri, Li Yi, Leonidas J. Guibas, and Hao Zhang. 2020. AdaCoSeg: Adaptive Shape Co-Segmentation With Group Consistency Loss. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
