“Learning to reconstruct botanical trees from single images” by Li, Kałużny, Klein, Michels, Pałubicki, et al. …
Conference:
Type(s):
Title:
- Learning to reconstruct botanical trees from single images
Session/Category Title: Natural Phenomena
Presenter(s)/Author(s):
Abstract:
We introduce a novel method for reconstructing the 3D geometry of botanical trees from single photographs. Faithfully reconstructing a tree from single-view sensor data is a challenging and open problem because many possible 3D trees exist that fit the tree’s shape observed from a single view. We address this challenge by defining a reconstruction pipeline based on three neural networks. The networks simultaneously mask out trees in input photographs, identify a tree’s species, and obtain its 3D radial bounding volume – our novel 3D representation for botanical trees. Radial bounding volumes (RBV) are used to orchestrate a procedural model primed on learned parameters to grow a tree that matches the main branching structure and the overall shape of the captured tree. While the RBV allows us to faithfully reconstruct the main branching structure, we use the procedural model’s morphological constraints to generate realistic branching for the tree crown. This constraints the number of solutions of tree models for a given photograph of a tree. We show that our method reconstructs various tree species even when the trees are captured in front of complex backgrounds. Moreover, although our neural networks have been trained on synthetic data with data augmentation, we show that our pipeline performs well for real tree photographs. We evaluate the reconstructed geometries with several metrics, including leaf area index and maximum radial tree distances.
References:
1. F. Anastacio, M. C. Sousa, F. Samavati, and J. A. Jorge. 2006. Modeling Plant Structures Using Concept Sketches. In Proceedings of the 4th International Symposium on Non-Photorealistic Animation and Rendering (NPAR ’06). Association for Computing Machinery, 105–113.
2. M. Aono and T.L. Kunii. 1984. Botanical Tree Image Generation. IEEE Comput. Graph. Appl. 4(5) (1984), 10–34.
3. O. Argudo, A. Chica, and C. Andujar. 2016. Single-picture Reconstruction and Rendering of Trees for Plausible Vegetation Synthesis. Comput. Graph. 57, C (2016), 55–67.
4. S. Behrendt, C. Colditz, O. Franzke, J. Kopf, and O. Deussen. 2005. Realistic real-time rendering of landscapes using billboard clouds. Comp. Graph. Forum 24, 3 (2005), 507–516.
5. B. Benes and E. U. Millán. 2002. Virtual Climbing Plants Competing for Space. In Proceedings of the Computer Animation (CA ’02). IEEE Computer Society, USA, 33.
6. W. Blozan. 2006. Tree Measuring Guidelines of the Eastern Native Tree Society. (2006).
7. D. Bradley, D. Nowrouzezahrai, and P. Beardsley. 2013. Image-based Reconstruction and Synthesis of Dense Foliage. ACM Trans. on Grap. 32, 4, Article 74 (2013), 74:1–74:10 pages.
8. L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. 2016. DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs. CoRR abs/1606.00915 (2016).
9. X. Chen, B. Neubert, Y.-Q. Xu, O. Deussen, and S. B. Kang. 2008. Sketch-Based Tree Modeling Using Markov Random Field. ACM Trans. on Grap. 27, 5, Article 109 (Dec. 2008), 9 pages.
10. O. Deussen, C. Colditz, M. Stamminger, and G. Drettakis. 2002. Interactive Visualization of Complex Plant Ecosystems. VIS ’02 (2002), 219–226.
11. N. Greene. 1989. Voxel Space Automata: Modeling with Stochastic Growth Processes in Voxel Space. SIGGRAPH Comp. Graph. 23, 3 (1989), 175–184.
12. J. Guo, H. Jiang, B. Benes, O. Deussen, X. Zhang, D. Lischinski, and H. Huang. 2020. Inverse Procedural Modeling of Branching Structures by Inferring L-Systems. ACM Trans. on Grap. 39, 5, Article 155 (June 2020), 13 pages.
13. R. Habel, A. Kusternig, and M. Wimmer. 2009. Physically Guided Animation of Trees. Comp. Graph. Forum 28, 2 (2009), 523–532.
14. T. Hädrich, B. Benes, O. Deussen, and S. Pirk. 2017. Interactive Modeling and Authoring of Climbing Plants. Comput. Graph. Forum 36, 2 (2017), 49–61.
15. T. Hastie, R. Tibshirani, and J. Friedman. 2001. The Elements of Statistical Learning. Springer New York Inc., New York, NY, USA.
16. H. Honda. 1971. Description of the form of trees by the parameters of the tree-like body: Effects of the branching angle and the branch length on the shape of the tree-like body. Journal of Theoretical Biology 31, 2 (1971), 331 — 338.
17. T. Ijiri, S. Owada, and T. Igarashi. 2006. Seamless Integration of Initial Sketching and Subsequent Detail Editing in Flower Modeling. Comp. Graph. Forum 25, 3 (2006), 617–624.
18. B. Karis. 2013. Real Shading in Unreal Engine 4. Technical Report. Epic Games. Y. Kawaguchi. 1982. A Morphological Study of the Form of Nature. SIGGRAPH Comp. Graph. 16, 3 (1982), 223–232.
19. J. Kratt, Mark Spicker, A. Guayaquil, M. Fišer, S. Pirk, O. Deussen, J. C. Hart, and B. Benes. 2015. Woodification: User-Controlled Cambial Growth Modeling. CGF 34, 2 (2015), 361–372.
20. A. Krizhevsky, I. Sutskever, and G. E. Hinton. 2012. ImageNet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems, Vol. 25. Curran Associates, Inc., 1097–1105.
21. T. Kugelstadt and E. Schoemer. 2016. Position and Orientation Based Cosserat Rods. In Proceedings of the 2016 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. Eurographics Association.
22. C. Li, O. Deussen, Y.-Z. Song, P. Willis, and P. Hall. 2011. Modeling and Generating Moving Trees from Video. ACM Trans. on Grap. 30, 6, Article 127 (2011), 127:1–127:12 pages.
23. Y. Li, X. Fan, N. J. Mitra, D. Chamovitz, D. Cohen-Or, and B. Chen. 2013. Analyzing Growing Plants from 4D Point Cloud Data. ACM Trans. on Grap. 32, 6, Article 157 (2013), 10 pages.
24. B. Lintermann and O. Deussen. 1999. Interactive Modeling of Plants. IEEE Comput. Graph. Appl. 19, 1 (1999), 56–65.
25. Y. Livny, S. Pirk, Z. Cheng, F. Yan, O. Deussen, D. Cohen-Or, and B. Chen. 2011. Texture-lobes for Tree Modelling. ACM Trans. on Grap. 30, 4, Article 53 (2011), 10 pages.
26. S. Longay, A. Runions, F. Boudon, and P. Prusinkiewicz. 2012. TreeSketch: interactive procedural modeling of trees on a tablet. In Proc. of the Intl. Symp. on SBIM. 107–120.
27. D. L. Michels, J. P. T. Mueller, and G. A. Sobottka. 2015. A physically based approach to the accurate simulation of stiff fibers and stiff fiber meshes. Comput. Graph. 53 (2015), 136–146.
28. R. Měch and P. Prusinkiewicz. 1996. Visual models of plants interacting with their environment. In Proc. of SIGGRAPH. ACM, 397–410.
29. B. Neubert, T. Franken, and O. Deussen. 2007. Approximate Image-based Tree-modeling Using Particle Flows. ACM Trans. on Grap. 26, 3, Article 88 (2007).
30. B. Neubert, S. Pirk, O. Deussen, and C. Dachsbacher. 2011. Improved Model- and View-Dependent Pruning of Large Botanical Scenes. Comp. Graph. Forum 30, 6 (2011), 1708–1718.
31. M. Okabe, S. Owada, and T. Igarashi. 2007. Interactive Design of Botanical Trees Using Freehand Sketches and Example-based Editing. In ACM SIGGRAPH Courses. ACM, Article 26.
32. P. E. Oppenheimer. 1986. Real time design and animation of fractal plants and trees. Proc. of SIGGRAPH 20, 4 (1986), 55–64.
33. W. Palubicki, K. Horel, S. Longay, A. Runions, B. Lane, R. Měch, and P. Prusinkiewicz. 2009. Self-organizing Tree Models for Image Synthesis. ACM Trans. on Grap. 28, 3, Article 58 (2009), 10 pages.
34. S. Pirk, B. Benes, T. Ijiri, Y. Li, O. Deussen, B. Chen, and R. Měch. 2016. Modeling Plant Life in Computer Graphics. In ACM SIGGRAPH 2016 Courses. Article 18, 180 pages.
35. S. Pirk, M. Jarząbek, T. Hädrich, D. L. Michels, and W. Palubicki. 2017. Interactive Wood Combustion for Botanical Tree Models. ACM Trans. on Grap. 36, 6, Article 197 (Nov. 2017), 12 pages.
36. S. Pirk, T. Niese, O. Deussen, and B. Neubert. 2012a. Capturing and animating the morphogenesis of polygonal tree models. ACM Trans. on Grap. 31, 6, Article 169 (2012), 10 pages.
37. S. Pirk, T. Niese, T. Hädrich, B. Benes, and O. Deussen. 2014. Windy Trees: Computing Stress Response for Developmental Tree Models. ACM Trans. on Grap. 33, 6, Article 204 (2014), 11 pages.
38. S. Pirk, O. Stava, J. Kratt, M. A. M. Said, B. Neubert, R. Měch, B. Benes, and O. Deussen. 2012b. Plastic trees: interactive self-adapting botanical tree models. ACM Trans. on Grap. 31, 4, Article 50 (2012), 10 pages.
39. P. Prusinkiewicz and Aristid Lindenmayer. 1990. The Algorithmic Beauty of Plants. Springer-Verlag New York, Inc.
40. L. Quan, P. Tan, G. Zeng, L. Yuan, J. Wang, and S. B. Kang. 2006. Image-Based Plant Modeling. ACM Trans. on Grap. 25, 3 (July 2006), 599–604.
41. E. Quigley, Y. Yu, J. Huang, W. Lin, and R. Fedkiw. 2018. Real-Time Interactive Tree Animation. IEEE Trans. on Vis. and Comp. Graphics 24, 5 (2018), 1717–1727.
42. A. Reche-Martinez, I. Martin, and G. Drettakis. 2004. Volumetric reconstruction and interactive rendering of trees from photographs. ACM Trans. on Grap. 23, 3 (2004), 720–727.
43. W. T. Reeves and R. Blau. 1985. Approximate and Probabilistic Algorithms for Shading and Rendering Structured Particle Systems. SIGGRAPH Comput. Graph. 19, 3 (July 1985), 313–322.
44. S. Ren, K. He, R. Girshick, and J. Sun. 2015. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. In Advances in Neural Information Processing Systems, C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (Eds.), Vol. 28. Curran Associates, Inc.
45. A. Runions, B. Lane, and P. Prusinkiewicz. 2007. Modeling Trees with a Space Colonization Algorithm. EG Nat. Phenom. (2007), 63–70.
46. H. Shao, T. Kugelstadt, T. Hädrich, W. Pałubicki, J. Bender, S. Pirk, and D. L. Michels. 2021. Accurately Solving Physical Systems with Graph Learning.
47. A. R. Smith. 1984. Plants, fractals, and formal languages. In Proc. of SIGGRAPH. ACM Press, 1–10.
48. O. Stava, S. Pirk, J. Kratt, B. Chen, R. Měch, O. Deussen, and B. Benes. 2014. Inverse Procedural Modelling of Trees. Comp. Graph. Forum 33, 6 (2014), 118–131.
49. P. Tan, T. Fang, J. Xiao, P. Zhao, and L. Quan. 2008. Single Image Tree Modeling. ACM Trans. on Grap. 27, 5, Article 108 (2008), 7 pages.
50. P. Tan, G. Zeng, J. Wang, S. B. Kang, and L. Quan. 2007. Image-based Tree Modeling. ACM Trans. on Grap. 26, 3, Article 87 (2007).
51. B. Wang, Y. Zhao, and J. Barbič. 2017b. Botanical Materials Based on Biomechanics. ACM Trans. on Grap. 36, 4, Article 135 (July 2017), 13 pages.
52. G. Wang, H. Laga, J. Jia, N. Xie, and H. Tabia. 2018a. Statistical Modeling of the 3D Geometry and Topology of Botanical Trees. Comp. Graph. Forum 37, 5 (2018), 185–198.
53. G. Wang, H. Laga, N. Xie, J. Jia, and H. Tabia. 2018b. The Shape Space of 3D Botanical Tree Models. ACM Trans. on Grap. 37, 1, Article 7 (Jan. 2018), 18 pages.
54. Y. Wang, X. Xue, X. Jin, and Z. Deng. 2017a. Creative Virtual Tree Modeling Through Hierarchical Topology-Preserving Blending. IEEE Trans. on Vis. and Comp. Graphics 23, 12 (2017), 2521–2534.
55. D. J. Watson. 1947. Comparative Physiological Studies on the Growth of Field Crops: I. Variation in Net Assimilation Rate and Leaf Area between Species and Varieties, and within and between Years. Annals of Botany 11, 1 (01 1947), 41–76.
56. J. Wither, F. Boudon, M.-P. Cani, and C. Godin. 2009. Structure from silhouettes: a new paradigm for fast sketch-based design of trees. Comp. Graph. Forum 28, 2 (2009), 541–550.
57. H. Xu, N. Gossett, and B. Chen. 2007. Knowledge and heuristic-based modeling of laser-scanned trees. ACM Trans. on Grap. 26, 4 (2007), Article 19, 13 pages.
58. Y. Zhao and J. Barbič. 2013. Interactive Authoring of Simulation-ready Plants. ACM Trans. on Grap. 32, 4, Article 84 (2013), 12 pages.
