“TreePartNet: neural decomposition of point clouds for 3D tree reconstruction” by Liu, Guo, Benes, Deussen, Zhang, et al. …
Conference:
Type(s):
Title:
- TreePartNet: neural decomposition of point clouds for 3D tree reconstruction
Session/Category Title: Natural Phenomena
Presenter(s)/Author(s):
Abstract:
We present TreePartNet, a neural network aimed at reconstructing tree geometry from point clouds obtained by scanning real trees. Our key idea is to learn a natural neural decomposition exploiting the assumption that a tree comprises locally cylindrical shapes. In particular, reconstruction is a two-step process. First, two networks are used to detect priors from the point clouds. One detects semantic branching points, and the other network is trained to learn a cylindrical representation of the branches. In the second step, we apply a neural merging module to reduce the cylindrical representation to a final set of generalized cylinders combined by branches. We demonstrate results of reconstructing realistic tree geometry for a variety of input models and with varying input point quality, e.g., noise, outliers, and incompleteness. We evaluate our approach extensively by using data from both synthetic and real trees and comparing it with alternative methods.
References:
1. Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. 2018. Learning Representations and Generative Models For 3D Point Clouds. In International Conference on Machine Learning (ICML), Vol. 80. 40–49.
2. Elie Aljalbout, V. Golkov, Yawar Siddiqui, and D. Cremers. 2018. Clustering with Deep Learning: Taxonomy and New Methods. ArXiv abs/1801.07648 (2018).
3. Oscar Argudo, Antonio Chica, and Carlos Andujar. 2016. Single-picture reconstruction and rendering of trees for plausible vegetation synthesis. Computers & Graphics 57 (2016), 55–67.
4. Matan Atzmon and Yaron Lipman. 2020. SAL: Sign Agnostic Learning of Shapes From Raw Data. In IEEE Computer Vision and Pattern Recognition (CVPR). 2565–2574.
5. Matan Atzmon, Haggai Maron, and Yaron Lipman. 2018. Point convolutional neural networks by extension operators. ACM Trans. on Graphics 37, 4, Article 71 (2018).
6. Bedrich Benes, Nathan Andrysco, and Ondřej Št’ava. 2009. Interactive modeling of virtual ecosystems. In Proc. of the Eurogr. Conference on Natural Phenomena. 9–16.
7. Derek Bradley, Derek Nowrouzezahrai, and Paul Beardsley. 2013. Image-based reconstruction and synthesis of dense foliage. ACM Trans. on Graphics (SIGGRAPH) 32, 4 (2013), 74.
8. Stéphane Calderon and Tamy Boubekeur. 2017. Bounding proxies for shape approximation. ACM Trans. on Graphics 36, 4 (2017), 57.
9. Junjie Cao, Andrea Tagliasacchi, Matt Olson, Hao Zhang, and Zhinxun Su. 2010. Point cloud skeletons via laplacian based contraction. In Shape Modeling International Conference. 187–197.
10. Zhiqin Chen, Andrea Tagliasacchi, and Hao Zhang. 2020. Bsp-net: Generating compact meshes via binary space partitioning. In IEEE Computer Vision and Pattern Recognition (CVPR). 45–54.
11. Zhiqin Chen and Hao Zhang. 2019. Learning implicit fields for generative shape modeling. In IEEE Computer Vision and Pattern Recognition (CVPR). 5939–5948.
12. Phillippe de Reffye, Claude Edelin, Jean Françon, Marc Jaegerl, and Claude Puech. 1988. Plant models faithful to botanical structure and development. ACM SIGGRAPH 22 (1988), 151–158.
13. Boyang Deng, Kyle Genova, Soroosh Yazdani, Sofien Bouaziz, Geoffrey Hinton, and Andrea Tagliasacchi. 2020. Cvxnet: Learnable convex decomposition. In IEEE Computer Vision and Pattern Recognition (CVPR). 31–44.
14. Shenglan Du, Roderik Lindenbergh, Hugo Ledoux, Jantien Stoter, and Liangliang Nan. 2019. AdTree: Accurate, Detailed, and Automatic Modelling of Laser-Scanned Trees. Remote Sensing 11, 18 (2019), 2074.
15. Philipp Erler, Paul Guerrero, Stefan Ohrhallinger, Niloy J. Mitra, and Michael Wimmer. 2020. Points2Surf: Learning Implicit Surfaces from Point Clouds. In European Conference on Computer Vision (ECCV), Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (Eds.). 108–124.
16. Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu. 1996. A density-based algorithm for discovering clusters in large spatial databases with noise. In KDD. 226–231.
17. Haoqiang Fan, Hao Su, and Leonidas J Guibas. 2017. A point set generation network for 3d object reconstruction from a single image. In IEEE Computer Vision and Pattern Recognition (CVPR). 605–613.
18. Keinosuke Fukunaga and Larry Hostetler. 1975. The estimation of the gradient of a density function, with applications in pattern recognition. IEEE Transactions on information theory 21, 1 (1975), 32–40.
19. Matheus Gadelha, Aruni RoyChowdhury, Gopal Sharma, Evangelos Kalogerakis, Liangliang Cao, Erik Learned-Miller, Rui Wang, and Subhransu Maji. 2020. Label-Efficient Learning on Point Clouds using Approximate Convex Decompositions. In European Conference on Computer Vision (ECCV). 473–491.
20. Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, and Thomas Funkhouser. 2020. Local Deep Implicit Functions for 3D Shape. In IEEE Computer Vision and Pattern Recognition (CVPR). 4857–4866.
21. Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sarna, William T Freeman, and Thomas Funkhouser. 2019. Learning Shape Templates with Structured Implicit Functions. In IEEE International Conference on Computer Vision (ICCV). 7153–7163.
22. Manish Goyal, Sundar Murugappan, Cecil Piya, William Benjamin, Yi Fang, Min Liu, and Karthik Ramani. 2012. Towards locally and globally shape-aware reverse 3D modeling. Computer-Aided Design 44, 6 (2012), 537–553.
23. Paul Guerrero, Yanir Kleiman, Maks Ovsjanikov, and Niloy J Mitra. 2018. PCPNet learning local shape properties from raw point clouds. Comp. Graph. Forum (Proc. EUROGRAPHICS) 37, 2 (2018), 75–85.
24. Jianwei Guo, Haiyong Jiang, Bedrich Benes, Oliver Deussen, Xiaopeng Zhang, Dani Lischinski, and Hui Huang. 2020. Inverse Procedural Modeling of Branching Structures by Inferring L-Systems. ACM Trans. on Graphics 39, 5 (2020), 1–13.
25. Jianwei Guo, Shibiao Xu, Dong-Ming Yan, Zhanglin Cheng, Marc Jaeger, and Xiaopeng Zhang. 2018. Realistic procedural plant modeling from multiple view images. IEEE Trans. on Vis. and Comp. Graphics 26, 2 (2018), 1372–1384.
26. Meng-Hao Guo, Jun-Xiong Cai, Zheng-Ning Liu, Tai-Jiang Mu, Ralph R. Martin, and Shi-Min Hu. 2021. PCT: Point cloud transformer. Computational Visual Media 7, 2 (Apr 2021), 187–199.
27. Rana Hanocka, Gal Metzer, Raja Giryes, and Daniel Cohen-Or. 2020. Point2Mesh: A Self-Prior for Deformable Meshes. ACM Trans. Graph. 39, 4, Article 126 (2020).
28. Pedro Hermosilla, Tobias Ritschel, Pere-Pau Vázquez, Àlvar Vinacua, and Timo Ropinski. 2018. Monte Carlo convolution for learning on non-uniformly sampled point clouds. In ACM Trans. on Graphics (SIGGRAPH Asia). ACM, 235.
29. Amir Hertz, Rana Hanocka, Raja Giryes, and Daniel Cohen-Or. 2020. PointGMM: a Neural GMM Network for Point Clouds. In IEEE Computer Vision and Pattern Recognition (CVPR). 12054–12063.
30. Hisao Honda. 1971. Description of the form of trees by the parameters of the tree-like body: Effects of the branching angle and the branch length on the shape of the tree-like body. Journal of Theoretical Biology 31, 2 (1971), 331 — 338.
31. Hui Huang, Shihao Wu, Daniel Cohen-Or, Minglun Gong, Hao Zhang, Guiqing Li, and Baoquan Chen. 2013. L1-medial skeleton of point cloud. ACM Trans. on Graphics 32, 4 (2013), 65–1.
32. Takahiro Isokane, Fumio Okura, Ayaka Ide, Yasuyuki Matsushita, and Yasushi Yagi. 2018. Probabilistic plant modeling via multi-view image-to-image translation. In IEEE Computer Vision and Pattern Recognition (CVPR). 2906–2915.
33. Vijai Jayadevan, Edward J. Delp, and Zygmunt Pizlo. 2019. Skeleton Extraction from 3D Point Clouds by Decomposing the Object into Parts. CoRR abs/1912.11932 (2019), 1–24. arXiv:1912.11932 http://arxiv.org/abs/1912.11932
34. Adrien Kaiser, Jose Alonso Ybanez Zepeda, and Tamy Boubekeur. 2019. A survey of simple geometric primitives detection methods for captured 3D data. Comp. Graph. Forum 38, 1 (2019), 167–196.
35. Angjoo Kanazawa, Shubham Tulsiani, Alexei A Efros, and Jitendra Malik. 2018. Learning category-specific mesh reconstruction from image collections. In European Conference on Computer Vision (ECCV). 371–386.
36. Michael Kazhdan, Matthew Bolitho, and Hugues Hoppe. 2006. Poisson surface reconstruction. In Proceedings of the fourth Eurographics symposium on Geometry processing, Vol. 7. 61–70.
37. Michael Kazhdan and Hugues Hoppe. 2013. Screened poisson surface reconstruction. ACM Trans. on Graphics 32, 3 (2013), 1–13.
38. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
39. Yann LeCun and Yoshua Bengio. 1998. Convolutional Networks for Images, Speech, and Time Series. MIT Press, Cambridge, MA, USA, 255–258.
40. Bosheng Li, Jacek Kałużny, Jonathan Klein, Dominik Michels, Wojtek Pałubicki, Bedrich Benes, and Sören Pirk. 2021. Learning to Reconstruct Botanical Trees from Single Images. ACM Trans. on Graphics 40, 6 (2021).
41. Chuan Li, Oliver Deussen, Yi-Zhe Song, Phil Willis, and Peter Hall. 2011. Modeling and Generating Moving Trees from Video. ACM Trans. on Graphics (SIGGRAPH Asia) 30, 6, Article 127 (2011), 12 pages.
42. Xuetao Li, Tong Wing Woon, Tiow Seng Tan, and Zhiyong Huang. 2001. Decomposing polygon meshes for interactive applications. In Proceedings of symposium on Interactive 3D graphics. 35–42.
43. Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. 2018. Pointcnn: Convolution on x-transformed points. In Advances in Neural Information Processing Systems. 820–830.
44. Chen-Hsuan Lin, Chen Kong, and Simon Lucey. 2018. Learning efficient point cloud generation for dense 3d object reconstruction. In Proc. AAAI Conference on Artificial Intelligence, Vol. 32. 7114–7121.
45. Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 2017. Focal loss for dense object detection. In IEEE International Conference on Computer Vision (ICCV). 2980–2988.
46. Aristid Lindenmayer. 1968. Mathematical models for cellular interaction in development. Journal of Theoretical Biology Parts I and II, 18 (1968), 280–315.
47. Zhihao Liu, Kai Wu, Jianwei Guo, Yunhai Wang, Oliver Deussen, and Zhanglin Cheng. 2021. Single Image Tree Reconstruction via Adversarial Network. Graphical Models 117 (2021), 101–115.
48. Marco Livesu, Nicholas Vining, Alla Sheffer, James Gregson, and Riccardo Scateni. 2013. PolyCut: monotone graph-cuts for PolyCube base-complex construction. ACM Trans. on Graphics 32, 6 (2013), 1–12.
49. Yotam Livny, Sören Pirk, Zhanglin Cheng, Feilong Yan, Oliver Deussen, Daniel Cohen-Or, and Baoquan Chen. 2011. Texture-lobes for tree modeling. In ACM Trans. on Graphics (SIGGRAPH). 1.
50. Yotam Livny, Feilong Yan, Matt Olson, Baoquan Chen, Hao Zhang, and Jihad El-Sana. 2010. Automatic reconstruction of tree skeletal structures from point clouds. ACM Trans. on Graphics (SIGGRAPH) 29, 6 (2010), 151.
51. Zhaoliang Lun, Matheus Gadelha, Evangelos Kalogerakis, Subhransu Maji, and Rui Wang. 2017. 3d shape reconstruction from sketches via multi-view convolutional networks. In 2017 International Conference on 3D Vision (3DV). IEEE, 67–77.
52. Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. 2019. Occupancy networks: Learning 3d reconstruction in function space. In IEEE Computer Vision and Pattern Recognition (CVPR). 4460–4470.
53. Boris Neubert, Thomas Franken, and Oliver Deussen. 2007. Approximate image-based tree-modeling using particle flows. In ACM SIGGRAPH 2007 papers. 88–es.
54. Andrew Y. Ng, Michael I. Jordan, and Yair Weiss. 2001. On spectral clustering: Analysis and an algorithm. In Advances in Neural Information Proc. Systems. 1998, 849–856.
55. Wojciech Palubicki, Kipp Horel, Steven Longay, Adam Runions, Radomír Měch, and Przemyslaw Prusinkiewicz. 2009. Self-organizing tree models for image synthesis. ACM Trans. on Graphics (SIGGRAPH) 28 (2009), 58:1–58:10. Issue 3.
56. Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Love-grove. 2019. DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation. In IEEE Computer Vision and Pattern Recognition (CVPR). 165–174.
57. Quang-Hieu Pham, Thanh Nguyen, Binh-Son Hua, Gemma Roig, and Sai-Kit Yeung. 2019. Jsis3d: Joint semantic-instance segmentation of 3d point clouds with multi-task pointwise networks and multi-value conditional random fields. In IEEE Computer Vision and Pattern Recognition (CVPR). 8827–8836.
58. Przemyslaw Prusinkiewicz. 1986. Graphical Applications of L-systems. In Proceedings on Graphics Interface/Vision Interface ’86. Canadian Inf. Proc. Society, 247–253.
59. Przemyslaw Prusinkiewicz and Aristid Lindenmayer. 1990. The Algorithmic Beauty of Plants. Springer-Verlag New York, Inc., New York, USA.
60. Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. 2017a. Pointnet: Deep learning on point sets for 3d classification and segmentation. In IEEE Computer Vision and Pattern Recognition (CVPR). 652–660.
61. Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. 2017b. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In Advances in Neural Information Processing Systems. 5099–5108.
62. Long Quan, Ping Tan, Gang Zeng, Lu Yuan, Jingdong Wang, and Sing Bing Kang. 2006. Image-based plant modeling. ACM Trans. on Graphics (SIGGRAPH) 25, 3 (2006), 599–604.
63. Alex Reche-Martinez, Ignacio Martin, and George Drettakis. 2004. Volumetric Reconstruction and Interactive Rendering of Trees from Photographs. ACM Trans. on Graphics (SIGGRAPH) 23, 3 (2004), 720–727.
64. Adam Runions, Brendan Lane, and Przemyslaw Prusinkiewicz. 2007. Modeling Trees with a Space Colonization Algorithm. In Proc. of the Third EG Conf. on Nat. Phenomena. 63–70.
65. Ilya Shlyakhter, Max Rozenoer, Julie Dorsey, and Seth Teller. 2001. Reconstructing 3D Tree Models from Instrumented Photographs. IEEE Comput. Graph. Appl. 21, 3 (May 2001), 53–61.
66. Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhofer. 2019. Deepvoxels: Learning persistent 3d feature embeddings. In IEEE Computer Vision and Pattern Recognition (CVPR). 2437–2446.
67. Ondrej Stava, B. Benes, Radomír Měch, D. G. Aliaga, and P. Kristof. 2010. Inverse Procedural Modeling by Automatic Generation of L-systems. Comp. Graph. Forum (Proc. EUROGRAPHICS) 29, 2 (2010), 665–674.
68. Ondrej Stava, Sören Pirk, Julian Kratt, Baoquan Chen, Radomir Mech, Oliver Deussen, and Bedrich Benes. 2014. Inverse Procedural Modelling of Trees. Comp. Graph. Forum 33, 6 (2014), 118–131.
69. Ping Tan, Tian Fang, Jianxiong Xiao, Peng Zhao, and Long Quan. 2008. Single Image Tree Modeling. ACM Trans. on Graphics (SIGGRAPH Asia) 27, 5, Article 108 (2008), 108:1–108:7 pages.
70. Ping Tan, Gang Zeng, Jingdong Wang, Sing Bing Kang, and Long Quan. 2007. Image-based tree modeling. ACM Trans. on Graphics (SIGGRAPH) 26, 3 (2007), 87.
71. Jiapeng Tang, Xiaoguang Han, Junyi Pan, Kui Jia, and Xin Tong. 2019. A Skeleton-bridged Deep Learning Approach for Generating Meshes of Complex Topologies from Single RGB Images. In IEEE Computer Vision and Pattern Recognition (CVPR). 4541–4550.
72. Maxim Tatarchenko, Alexey Dosovitskiy, and Thomas Brox. 2017. Octree generating networks: Efficient convolutional architectures for high-resolution 3d outputs. In IEEE International Conference on Computer Vision (ICCV). 2088–2096.
73. Shubham Tulsiani, Hao Su, Leonidas J Guibas, Alexei A Efros, and Jitendra Malik. 2017. Learning shape abstractions by assembling volumetric primitives. In IEEE Computer Vision and Pattern Recognition (CVPR). 2635–2643.
74. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems, Vol. 30. 5998–6008.
75. Giuseppe Vettigli. 2018. MiniSom: minimalistic and NumPy-based implementation of the Self Organizing Map. https://github.com/JustGlowing/minisom/
76. Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. 2018c. Pixel2mesh: Generating 3d mesh models from single rgb images. In European Conference on Computer Vision (ECCV). 52–67.
77. Weiyue Wang, Ronald Yu, Qiangui Huang, and Ulrich Neumann. 2018b. Sgpn: Similarity group proposal network for 3d point cloud instance segmentation. In IEEE Computer Vision and Pattern Recognition (CVPR). 2569–2578.
78. Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. 2018a. Dynamic graph cnn for learning on point clouds. ACM Trans. on Graphics 38, 5 (2018), 12 pages.
79. G B West, James H Brown, and B J Enquist. 1999. A general model for the structure and allometry of plant vascular systems. Nature 400, 6745 (1999), 664–667.
80. Erik Wijmans. 2018. Pointnet++ Pytorch. (2018). https://github.com/erikwijmans/Pointnet2_PyTorch
81. Peter Wohlleben. 2016. The hidden life of trees: What they feel, how they communicate—Discoveries from a secret world. Vol. 1. Greystone Books.
82. Wenxuan Wu, Zhongang Qi, and Li Fuxin. 2019. Pointconv: Deep convolutional networks on 3d point clouds. In IEEE Computer Vision and Pattern Recognition (CVPR). 9621–9630.
83. Junyuan Xie, Ross Girshick, and Ali Farhadi. 2016. Unsupervised deep embedding for clustering analysis. In ICML. 478–487.
84. Hui Xu, Nathan Gossett, and Baoquan Chen. 2007. Knowledge and heuristic-based modeling of laser-scanned trees. ACM Trans. on Graphics 26, 4 (2007), 19.
85. Zhan Xu, Yang Zhou, Evangelos Kalogerakis, Chris Landreth, and Karan Singh. 2020. RigNet: Neural Rigging for Articulated Characters. ACM Trans. on Graphics 39, 4, Article 58 (2020), 14 pages.
86. D. M. Yan, J. Wintz, B. Mourrain, W. Wang, F. Boudon, and C. Godin. 2009. Efficient and robust reconstruction of botanical branching structure from laser scanned points. In Proc. 11th IEEE Int. Conf. CAD/Graph. Comput. 572–575.
87. Xu Yan, Chaoda Zheng, Zhen Li, Sheng Wang, and Shuguang Cui. 2020. PointASNL: Robust Point Clouds Processing using Nonlocal Neural Networks with Adaptive Sampling. In IEEE Computer Vision and Pattern Recognition (CVPR). 5589–5598.
88. Lei Yi, Hongjun Li, Jianwei Guo, Oliver Deussen, and Xiaopeng Zhang. 2018. Tree growth modelling constrained by growth equations. Comp. Graph. Forum 37, 1 (2018), 239–253.
89. Kangxue Yin, Zhiqin Chen, Hui Huang, Daniel Cohen-Or, and Hao Zhang. 2019. LOGAN: Unpaired Shape Transform in Latent Overcomplete Space. ACM Trans. on Graphics (SIGGRAPH Asia) 38, 6 (2019), 198:1–198:13.
90. Kangxue Yin, Hui Huang, Daniel Cohen-Or, and Hao Zhang. 2018. P2P-NET: bidirectional point displacement net for shape transform. ACM Trans. on Graphics (SIGGRAPH Asia) 37, 4 (2018), 152.
91. Xiaopeng Zhang, Hongjun Li, Mingrui Dai, Wei Ma, and Long Quan. 2014. Data-driven synthetic modeling of trees. IEEE Trans. on Vis. and Comp. Graphics 20, 9 (2014), 1214–1226.
92. Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip Torr, and Vladlen Koltun. 2020. Point transformer. In arXiv preprint arXiv:2012.09164. 1–10.
93. Yang Zhou, Kangxue Yin, Hui Huang, Hao Zhang, Minglun Gong, and Daniel Cohen-Or. 2015. Generalized cylinder decomposition. ACM Trans. Graph. 34, 6 (2015), 171–1.
