“SymmetryNet: learning to predict reflectional and rotational symmetries of 3D shapes from single-view RGB-D images” by Shi, Huang, Zhang, Xu, Rusinkiewicz, et al. …
Conference:
Type(s):
Title:
- SymmetryNet: learning to predict reflectional and rotational symmetries of 3D shapes from single-view RGB-D images
Session/Category Title: Generation and Inference from Images
Presenter(s)/Author(s):
Abstract:
We study the problem of symmetry detection of 3D shapes from single-view RGB-D images, where severely missing data renders geometric detection approach infeasible. We propose an end-to-end deep neural network which is able to predict both reflectional and rotational symmetries of 3D objects present in the input RGB-D image. Directly training a deep model for symmetry prediction, however, can quickly run into the issue of overfitting. We adopt a multi-task learning approach. Aside from symmetry axis prediction, our network is also trained to predict symmetry correspondences. In particular, given the 3D points present in the RGB-D image, our network outputs for each 3D point its symmetric counterpart corresponding to a specific predicted symmetry. In addition, our network is able to detect for a given shape multiple symmetries of different types. We also contribute a benchmark of 3D symmetry detection based on single-view RGB-D images. Extensive evaluation on the benchmark demonstrates the strong generalization ability of our method, in terms of high accuracy of both symmetry axis prediction and counterpart estimation. In particular, our method is robust in handling unseen object instances with large variation in shape, multi-symmetry composition, as well as novel object categories.
References:
1. Armen Avetisyan, Manuel Dahnert, Angela Dai, Manolis Savva, Angel X Chang, and Matthias Nießner. Scan2cad: Learning cad model alignment in rgb-d scans. In Proc. CVPR, pages 2614–2623, 2019.Google ScholarCross Ref
2. Martin Bokeloh, Alexander Berner, Michael Wand, H-P Seidel, and Andreas Schilling. Symmetry detection using feature lines. In Computer Graphics Forum, volume 28, pages 697–706, 2009.Google ScholarCross Ref
3. Berk Calli, Arjun Singh, Aaron Walsman, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M Dollar. The ycb object and model set: Towards common benchmarks for manipulation research. In International Conference on Advanced Robotics (ICAR), pages 510–517. IEEE, 2015.Google ScholarCross Ref
4. Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012, 2015.Google Scholar
5. Changhyun Choi and Henrik I Christensen. 3d pose estimation of daily objects using an rgb-d camera. In Proc. IROS, pages 3342–3349. IEEE, 2012.Google ScholarCross Ref
6. Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proc. CVPR, pages 5828–5839, 2017.Google ScholarCross Ref
7. Aleksandrs Ecins, Cornelia Fermüller, and Yiannis Aloimonos. Seeing behind the scene: Using symmetry to reason about objects in cluttered environments. In Proc. IROS, pages 7193–7200. IEEE, 2018.Google ScholarCross Ref
8. Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, et al. A density-based algorithm for discovering clusters in large spatial databases with noise. In KDD, volume 96, pages 226–231, 1996.Google ScholarDigital Library
9. Christopher Funk, Seungkyu Lee, Martin R Oswald, Stavros Tsogkas, Wei Shen, Andrea Cohen, Sven Dickinson, and Yanxi Liu. 2017 iccv challenge: Detecting symmetry in the wild. In Proc. ICCV, pages 1692–1701, 2017.Google Scholar
10. Lin Gao, Ling-Xiao Zhang, Hsien-Yu Meng, Yi-Hui Ren, Yu-Kun Lai, and Leif Kobbelt. Prs-net: Planar reflective symmetry detection net for 3d models. arXiv preprint arXiv:1910.06511, 2019.Google Scholar
11. Natasha Gelfand and Leonidas J Guibas. Shape segmentation using local slippage analysis. In Proceedings of the 2004 Eurographics/ACM SIGGRAPH symposium on Geometry processing, pages 214–223, 2004.Google ScholarDigital Library
12. Georgios Georgakis, Srikrishna Karanam, Ziyan Wu, and Jana Kosecka. Matching rgb images to cad models for object pose estimation. arXiv preprint arXiv:1811.07249, 2018.Google Scholar
13. Tomáš Hodaň, Xenophon Zabulis, Manolis Lourakis, Štěpán Obdržálek, and Jiří Matas. Detection and fine 3d pose estimation of texture-less objects in rgb-d images. In Proc. IROS, pages 4421–4428. IEEE, 2015.Google ScholarCross Ref
14. Qichang Hu, Huibing Wang, Teng Li, and Chunhua Shen. Deep cnns with spatially weighted pooling for fine-grained car recognition. IEEE Transactions on Intelligent Transportation Systems, 18(11):3147–3156, 2017.Google ScholarDigital Library
15. Haibin Huang, Evangelos Kalogerakis, Siddhartha Chaudhuri, Duygu Ceylan, Vladimir G Kim, and Ersin Yumer. Learning local shape descriptors from part correspondences with multiview convolutional networks. ACM Transactions on Graphics (TOG), 37(1):1–14, 2017.Google Scholar
16. Wei Ke, Jie Chen, Jianbin Jiao, Guoying Zhao, and Qixiang Ye. Srn: side-output residual network for object symmetry detection in the wild. In Proc. CVPR, pages 1068–1076, 2017.Google ScholarCross Ref
17. Vladimir G Kim, Yaron Lipman, and Thomas Funkhouser. Blended intrinsic maps. ACM Transactions on Graphics (TOG), 30(4):1–12, 2011.Google Scholar
18. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proc. ICLR, 2015.Google Scholar
19. Yoshinori Konishi, Kosuke Hattori, and Manabu Hashimoto. Real-time 6d object pose estimation on cpu. arXiv preprint arXiv:1811.08588, 2018.Google Scholar
20. Andreas Kuehnle. Symmetry-based recognition of vehicle rears. Pattern Recognition Letters, 12(4):249–258, 1991.Google ScholarDigital Library
21. Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1–2):83–97, 1955.Google Scholar
22. Bo Li, Henry Johan, Yuxiang Ye, and Yijuan Lu. Efficient view-based 3d reflection symmetry detection. In SIGGRAPH Asia 2014 Creative Shape Modeling and Design, pages 1–8. 2014.Google ScholarDigital Library
23. Jun Li, Kai Xu, Siddhartha Chaudhuri, Ersin Yumer, Hao Zhang, and Leonidas Guibas. Grass: Generative recursive autoencoders for shape structures. ACM Transactions on Graphics (TOG), 36(4):1–14, 2017.Google Scholar
24. Yaron Lipman, Xiaobai Chen, Ingrid Daubechies, and Thomas Funkhouser. Symmetry factored embedding and distance. ACM Transactions on Graphics (TOG), July 2010.Google ScholarDigital Library
25. Minghua Liu, Lu Sheng, Sheng Yang, Jing Shao, and Shi-Min Hu. Morphing and sampling network for dense point cloud completion. In Proc. AAAI, 2020.Google ScholarCross Ref
26. Yanxi Liu and Robert T Collins. A computational model for repeated pattern perception using frieze and wallpaper groups. In Proc. CVPR, volume 1, pages 537–544. IEEE, 2000.Google Scholar
27. Yanxi Liu and Robert T Collins. Skewed symmetry groups. In Proc. CVPR, volume 1, pages I-I. IEEE, 2001.Google Scholar
28. Yanxi Liu, Hagit Hel-Or, Craig S Kaplan, Luc Van Gool, et al. Computational symmetry in computer vision and computer graphics. Foundations and Trends® in Computer Graphics and Vision, 5(1–2):1–195, 2010.Google Scholar
29. Giovanni Marola. On the detection of the axes of symmetry of symmetric and almost symmetric planar images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 11(1):104–108, 1989.Google ScholarDigital Library
30. Aurélien Martinet, Cyril Soler, Nicolas Holzschuch, and François X Sillion. Accurate detection of symmetries in 3d shapes. ACM Transactions on Graphics (TOG), 25(2): 439–464, 2006.Google Scholar
31. Niloy J Mitra, Leonidas J Guibas, and Mark Pauly. Partial and approximate symmetry detection for 3d geometry. In ACM Transactions on Graphics (TOG), volume 25, pages 560–568. ACM, 2006.Google ScholarDigital Library
32. Niloy J Mitra, Mark Pauly, Michael Wand, and Duygu Ceylan. Symmetry in 3d geometry: Extraction and applications. In Computer Graphics Forum, volume 32, pages 1–23, 2013.Google ScholarDigital Library
33. Hideo Ogawa. Symmetry analysis of line drawings using the hough transform. Pattern Recognition Letters, 12(1):9–12, 1991.Google ScholarDigital Library
34. Maks Ovsjanikov, Jian Sun, and Leonidas Guibas. Global intrinsic symmetries of shapes. In Computer Graphics Forum, volume 27, pages 1341–1348, 2008.Google ScholarCross Ref
35. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, pages 8024–8035, 2019.Google Scholar
36. Sida Peng, Yuan Liu, Qixing Huang, Xiaowei Zhou, and Hujun Bao. PVNet: Pixel-wise voting network for 6dof pose estimation. In Proc. CVPR, pages 4561–4570, 2019.Google ScholarCross Ref
37. Joshua Podolak, Philip Shilane, Aleksey Golovinskiy, Szymon Rusinkiewicz, and Thomas Funkhouser. A planar-reflective symmetry transform for 3D shapes. ACM Transactions on Graphics (TOG), 25(3), July 2006.Google Scholar
38. Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proc. CVPR, pages 652–660, 2017.Google Scholar
39. D. Raviv, A. M. Bronstein, M. M. Bronstein, and R. Kimmel. Symmetries of non-rigid shapes. In Proc. ICCV, pages 1–7. IEEE, 2007.Google ScholarCross Ref
40. Dan Raviv, Alexander M Bronstein, Michael M Bronstein, and Ron Kimmel. Full and partial symmetries of non-rigid shapes. International Journal of Computer Vision, 89 (1):18–39, 2010.Google ScholarDigital Library
41. Wei Shen, Xiang Bai, Zihao Hu, and Zhijiang Zhang. Multiple instance subspace learning via partial random projection tree for local reflection symmetry in natural images. Pattern Recognition, 52:306–316, 2016a.Google ScholarDigital Library
42. Wei Shen, Kai Zhao, Yuan Jiang, Yan Wang, Zhijiang Zhang, and Xiang Bai. Object skeleton extraction in natural images by fusing scale-associated deep side outputs. In Proc. CVPR, pages 222–230, 2016b.Google ScholarCross Ref
43. Ching L Teo, Cornelia Fermuller, and Yiannis Aloimonos. Detection and segmentation of 2d curved reflection symmetric structures. In Proc. ICCV, pages 1644–1652, 2015.Google ScholarDigital Library
44. Stavros Tsogkas and Iasonas Kokkinos. Learning-based symmetry detection in natural images. In Proc. ECCV, pages 41–54. Springer, 2012.Google ScholarDigital Library
45. Oliver Van Kaick, Hao Zhang, Ghassan Hamarneh, and Daniel Cohen-Or. A survey on shape correspondence. In Computer Graphics Forum, volume 30, pages 1681–1707, 2011.Google Scholar
46. Chen Wang, Danfei Xu, Yuke Zhu, Roberto Martín-Martín, Cewu Lu, Li Fei-Fei, and Silvio Savarese. Densefusion: 6d object pose estimation by iterative dense fusion. In Proc. CVPR, pages 3343–3352, 2019a.Google ScholarCross Ref
47. He Wang, Srinath Sridhar, Jingwei Huang, Julien Valentin, Shuran Song, and Leonidas J Guibas. Normalized object coordinate space for category-level 6d object pose and size estimation. In Proc. CVPR, pages 2642–2651, 2019b.Google ScholarCross Ref
48. Yanzhen Wang, Kai Xu, Jun Li, Hao Zhang, Ariel Shamir, Ligang Liu, Zhiquan Cheng, and Yueshan Xiong. Symmetry hierarchy of man-made objects. In Computer Graphics Rorum, volume 30, pages 287–296, 2011.Google ScholarCross Ref
49. Lingyu Wei, Qixing Huang, Duygu Ceylan, Etienne Vouga, and Hao Li. Dense human body correspondences using convolutional networks. In Proc. CVPR, pages 1544–1553, 2016.Google ScholarCross Ref
50. Kai Xu, Hao Zhang, Andrea Tagliasacchi, Ligang Liu, Guo Li, Min Meng, and Yueshan Xiong. Partial intrinsic reflectional symmetry of 3d shapes. ACM Transactions on Graphics (TOG), 28(5):1–10, 2009.Google Scholar
51. Yaoqing Yang, Chen Feng, Yiru Shen, and Dong Tian. Foldingnet: Point cloud autoencoder via deep grid deformation. In Proc. CVPR, pages 206–215, 2018.Google Scholar
52. Raymond KK Yip. A hough transform technique for the detection of reflectional symmetry and skew-symmetry. Pattern Recognition Letters, 21(2):117–130, 2000.Google ScholarDigital Library
53. Fenggen Yu, Kun Liu, Yan Zhang, Chenyang Zhu, and Kai Xu. Partnet: A recursive part decomposition network for fine-grained and hierarchical shape segmentation. In Proc. CVPR, pages 9491–9500, 2019.Google ScholarCross Ref
54. Andy Zeng, Shuran Song, Matthias Nießner, Matthew Fisher, Jianxiong Xiao, and Thomas Funkhouser. 3dmatch: Learning local geometric descriptors from rgb-d reconstructions. In Proc. CVPR, pages 1802–1811, 2017.Google ScholarCross Ref
55. Thomas Zielke, Michael Brauckmann, and Werner von Seelen. Intensity and edge-based symmetry detection applied to car-following. In Proc. ECCV, pages 865–873. Springer, 1992.Google ScholarCross Ref
