“Fusion 360 gallery: a dataset and environment for programmatic CAD construction from human design sequences” by Willis, Pu, Luo, Chu, Du, et al. …

Parametric computer-aided design (CAD) is a standard paradigm used to design manufactured objects, where a 3D shape is represented as a program supported by the CAD software. Despite the pervasiveness of parametric CAD and a growing interest from the research community, currently there does not exist a dataset of realistic CAD models in a concise programmatic form. In this paper we present the Fusion 360 Gallery, consisting of a simple language with just the sketch and extrude modeling operations, and a dataset of 8,625 human design sequences expressed in this language. We also present an interactive environment called the Fusion 360 Gym, which exposes the sequential construction of a CAD program as a Markov decision process, making it amendable to machine learning approaches. As a use case for our dataset and environment, we define the CAD reconstruction task of recovering a CAD program from a target geometry. We report results of applying state-of-the-art methods of program synthesis with neurally guided search on this task.

1. Silvia Ansaldi, Leila De Floriani, and Bianca Falcidieno. 1985. Geometric modeling of solid objects by using a face adjacency graph representation. ACM SIGGRAPH Computer Graphics 19, 3 (1985), 131–139.Google ScholarDigital Library
2. Autodesk. 2012. Inventor Feature Recognition. https://apps.autodesk.com/INVNTOR/en/Detail/Index?id=9172877436288348979Google Scholar
3. Autodesk. 2014. Fusion 360 API. http://help.autodesk.com/view/fusion360/ENU/?guid=GUID-7B5A90C8-E94C-48DA-B16B-430729B734DCGoogle Scholar
4. Autodesk. 2015. Autodesk Online Gallery. https://gallery.autodesk.comGoogle Scholar
5. Pal Benko, Geza Kos, Pál Benkö, Laszlo Andor, Géza Kós, Tamas Varady, László Andor, and Ralph Martin. 2002. Constrained Fitting in Reverse Engineering.Google Scholar
6. Suzanne Fox Buchele. 2000. Three-dimensional binary space partitioning tree and constructive solid geometry tree construction from algebraic boundary representations. (2000).Google Scholar
7. Suzanne F Buchele and Richard H Crawford. 2003. Three-dimensional halfspace constructive solid geometry tree construction from implicit boundary representations. In Proceedings of the eighth ACM symposium on Solid modeling and applications. 135–144.Google ScholarDigital Library
8. Suzanne F Buchele and Angela C Roles. 2001. Binary space partitioning tree and constructive solid geometry representations for objects bounded by curved surfaces.. In CCCG. Citeseer, 49–52.Google Scholar
9. Francesco Buonamici, Monica Carfagni, Rocco Furferi, Lapo Governi, Alessandro Lapini, and Yary Volpe. 2018. Reverse engineering modeling methods and tools: a survey. Computer-Aided Design and Applications 15, 3 (2018), 443–464. arXiv:https://doi.org/10.1080/16864360.2017.1397894 Google ScholarCross Ref
10. Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. 2015. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012 (2015).Google Scholar
11. Zhiqin Chen, Andrea Tagliasacchi, and Hao Zhang. 2020. Bsp-net: Generating compact meshes via binary space partitioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 45–54.Google ScholarCross Ref
12. Dassault. 2019. Solidworks FeatureWorks. https://help.solidworks.com/2019/english/SolidWorks/fworks/c_Overview_of_FeatureWorks.htmGoogle Scholar
13. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition. Ieee, 248–255.Google ScholarCross Ref
14. Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju, Rishabh Singh, Abdel-rahman Mohamed, and Pushmeet Kohli. 2017. Robustfill: Neural program learning under noisy i/o. arXiv preprint arXiv:1703.07469 (2017).Google Scholar
15. Tao Du, Jeevana Priya Inala, Yewen Pu, Andrew Spielberg, Adriana Schulz, Daniela Rus, Armando Solar-Lezama, and Wojciech Matusik. 2018. Inversecsg: Automatic conversion of 3d models to csg trees. ACM Transactions on Graphics (TOG) 37, 6 (2018), 1–16.Google ScholarDigital Library
16. Mathias Eitz, James Hays, and Marc Alexa. 2012. How do humans sketch objects? ACM Transactions on graphics (TOG) 31, 4 (2012), 1–10.Google Scholar
17. Kevin Ellis, Maxwell Nye, Yewen Pu, Felix Sosa, Josh Tenenbaum, and Armando Solar-Lezama. 2019. Write, execute, assess: Program synthesis with a repl. In Advances in Neural Information Processing Systems. 9169–9178.Google Scholar
18. Kevin Ellis, Daniel Ritchie, Armando Solar-Lezama, and Josh Tenenbaum. 2018. Learning to Infer Graphics Programs from Hand-Drawn Images. In Advances in Neural Information Processing Systems, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.), Vol. 31. Curran Associates, Inc., 6059–6068. https://proceedings.neurips.cc/paper/2018/file/6788076842014c83cedadbe6b0ba0314-Paper.pdfGoogle Scholar
19. Pierre-Alain Fayolle and Alexander Pasko. 2016. An evolutionary approach to the extraction of object construction trees from 3D point clouds. Computer-Aided Designs 74 (2016), 1–17.Google ScholarDigital Library
20. Markus Friedrich, Pierre-Alain Fayolle, Thomas Gabor, and Claudia Linnhoff-Popien. 2019. Optimizing evolutionary CSG tree extraction. In Proceedings of the Genetic and Evolutionary Computation Conference. 1183–1191.Google ScholarDigital Library
21. Lin Gao, Jie Yang, Tong Wu, Yu-Jie Yuan, Hongbo Fu, Yu-Kun Lai, and Hao Zhang. 2019. SDM-NET: Deep generative network for structured deformable mesh. ACM Transactions on Graphics (TOG) 38, 6 (2019), 1–15.Google ScholarDigital Library
22. Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. 2017. Neural message passing for quantum chemistry. In International Conference on Machine Learning. PMLR, 1263–1272.Google Scholar
23. Yulia Gryaditskaya, Mark Sypesteyn, Jan Willem Hoftijzer, Sylvia Pont, Fredo Durand, and Adrien Bousseau. 2019. Opensketch: A richly-annotated dataset of product design sketches. ACM Transactions on Graphics (TOG) 38, 6 (2019), 232.Google ScholarDigital Library
24. Karim Hamza and Kazuhiro Saitou. 2004. Optimization of constructive solid geometry via a tree-based multi-objective genetic algorithm. In Genetic and Evolutionary Computation Conference. Springer, 981–992.Google ScholarCross Ref
25. Pradeep Kumar Jayaraman, Aditya Sanghi, Joseph Lambourne, Thomas Davies, Hooman Shayani, and Nigel Morris. 2020. UV-Net: Learning from Curve-Networks and Solids. arXiv preprint arXiv:2006.10211 (2020).Google Scholar
26. R. Kenny Jones, Theresa Barton, Xianghao Xu, Kai Wang, Ellen Jiang, Paul Guerrero, Niloy Mitra, and Daniel Ritchie. 2020. ShapeAssembly: Learning to Generate Programs for 3D Shape Structure Synthesis. ACM Transactions on Graphics (TOG), Siggraph Asia 2020 39, 6 (2020), Article 234.Google Scholar
27. Ashwin Kalyan, Abhishek Mohta, Oleksandr Polozov, Dhruv Batra, Prateek Jain, and Sumit Gulwani. 2018. Neural-guided deductive search for real-time program synthesis from examples. arXiv preprint arXiv:1804.01186 (2018).Google Scholar
28. Kacper Kania, Maciej Zięba, and Tomasz Kajdanowicz. 2020. UCSG-Net-Unsupervised Discovering of Constructive Solid Geometry Tree. arXiv preprint arXiv:2006.09102 (2020).Google Scholar
29. Sangpil Kim, Hyung-gun Chi, Xiao Hu, Qixing Huang, and Karthik Ramani. 2020. A Large-scale Annotated Mechanical Components Benchmark for Classification and Retrieval Tasks with Deep Neural Networks. In Proceedings of 16th European Conference on Computer Vision (ECCV).Google ScholarDigital Library
30. Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard Zemel. 2018. Neural relational inference for interacting systems. In International Conference on Machine Learning. PMLR, 2688–2697.Google Scholar
31. Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 (2016).Google Scholar
32. Sebastian Koch, Albert Matveev, Zhongshi Jiang, Francis Williams, Alexey Artemov, Evgeny Burnaev, Marc Alexa, Denis Zorin, and Daniele Panozzo. 2019. ABC: A big CAD model dataset for geometric deep learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 9601–9611.Google ScholarCross Ref
33. Changjian Li, Hao Pan, Adrien Bousseau, and Niloy J. Mitra. 2020b. Sketch2CAD: Sequential CAD Modeling by Sketching in Context. ACM Trans. Graph. (Proceedings of SIGGRAPH Asia 2020) 39, 6 (2020), 164:1–164:14. Google ScholarDigital Library
34. Jun Li, Chengjie Niu, and Kai Xu. 2020a. Learning part generation and assembly for structure-aware shape synthesis. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 11362–11369.Google ScholarCross Ref
35. Renjie Liao, Yujia Li, Yang Song, Shenlong Wang, Will Hamilton, David K Duvenaud, Raquel Urtasun, and Richard Zemel. 2019. Efficient graph generation with graph recurrent attention networks. In Advances in Neural Information Processing Systems. 4255–4265.Google Scholar
36. Cheng Lin, Tingxiang Fan, Wenping Wang, and Matthias Nießner. 2020. Modeling 3D Shapes by Reinforcement Learning. ECCV (2020).Google Scholar
37. Kaichun Mo, Paul Guerrero, Li Yi, Hao Su, Peter Wonka, Niloy J. Mitra, and Leonidas J. Guibas. 2019a. StructureNet: Hierarchical Graph Networks for 3D Shape Generation. ACM Trans. Graph. 38, 6, Article 242 (Nov. 2019), 19 pages. Google ScholarDigital Library
38. Kaichun Mo, Shilin Zhu, Angel X Chang, Li Yi, Subarna Tripathi, Leonidas J Guibas, and Hao Su. 2019b. Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 909–918.Google ScholarCross Ref
39. Chandrakana Nandi, Anat Caspi, Dan Grossman, and Zachary Tatlock. 2017. Programming language tools and techniques for 3D printing. In 2nd Summit on Advances in Programming Languages (SNAPL 2017). Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik.Google Scholar
40. Chandrakana Nandi, James R Wilcox, Pavel Panchekha, Taylor Blau, Dan Grossman, and Zachary Tatlock. 2018. Functional programming for compiling and decompiling computer-aided design. Proceedings of the ACM on Programming Languages 2, ICFP (2018), 1–31.Google ScholarDigital Library
41. Chandrakana Nandi, Max Willsey, Adam Anderson, James R. Wilcox, Eva Darulova, Dan Grossman, and Zachary Tatlock. 2020. Synthesizing Structured CAD Models with Equality Saturation and Inverse Transformations. In Proceedings of the 41st ACM SIGPLAN Conference on Programming Language Design and Implementation. 31–44.Google ScholarDigital Library
42. Charlie Nash, Yaroslav Ganin, SM Ali Eslami, and Peter Battaglia. 2020. Polygen: An autoregressive generative model of 3d meshes. In International Conference on Machine Learning. PMLR, 7220–7229.Google Scholar
43. Patsorn Sangkloy, Nathan Burnell, Cusuh Ham, and James Hays. 2016. The sketchy database: learning to retrieve badly drawn bunnies. ACM Transactions on Graphics (TOG) 35, 4 (2016), 1–12.Google ScholarDigital Library
44. Nadav Schor, Oren Katzir, Hao Zhang, and Daniel Cohen-Or. 2019. CompoNet: Learning to generate the unseen by part synthesis and composition. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 8759–8768.Google ScholarCross Ref
45. Adriana Schulz, Ariel Shamir, David I. W. Levin, Pitchaya Sitthi-Amorn, and Wojciech Matusik. 2014. Design and Fabrication by Example. ACM Transactions on Graphics (Proceedings SIGGRAPH 2014) 33, 4 (2014).Google ScholarDigital Library
46. Ari Seff, Yaniv Ovadia, Wenda Zhou, and Ryan P. Adams. 2020. SketchGraphs: A Large-Scale Dataset for Modeling Relational Geometry in Computer-Aided Design. In ICML 2020 Workshop on Object-Oriented Learning.Google Scholar
47. Jami J Shah, David Anderson, Yong Se Kim, and Sanjay Joshi. 2001. A discourse on geometric feature recognition from CAD models. J. Comput. Inf. Sci. Eng. 1, 1 (2001), 41–51.Google ScholarDigital Library
48. Vadim Shapiro and Donald L Vossler. 1993. Separation for boundary to CSG conversion. ACM Transactions on Graphics (TOG) 12, 1 (1993), 35–55.Google ScholarDigital Library
49. Gopal Sharma, Rishabh Goyal, Difan Liu, Evangelos Kalogerakis, and Subhransu Maji. 2017. CSGNet: Neural Shape Parser for Constructive Solid Geometry. CoRR abs/1712.08290 (2017). arXiv preprint arXiv:1712.08290 (2017).Google Scholar
50. Binil Starly. 2020. FabWave – 3D Part Repository. https://www.dimelab.org/fabwaveGoogle Scholar
51. O. Stava, S. Pirk, J. Kratt, B. Chen, R. Mundefinedch, O. Deussen, and B. Benes. 2014. Inverse Procedural Modelling of Trees. Comput. Graph. Forum 33, 6 (Sept. 2014), 118–131. Google ScholarDigital Library
52. Minhyuk Sung, Hao Su, Vladimir G Kim, Siddhartha Chaudhuri, and Leonidas Guibas. 2017. ComplementMe: Weakly-supervised component suggestions for 3D modeling. ACM Transactions on Graphics (TOG) 36, 6 (2017), 1–12.Google ScholarDigital Library
53. Jerry O. Talton, Yu Lou, Steve Lesser, Jared Duke, Radomír Měch, and Vladlen Koltun. 2011. Metropolis Procedural Modeling. ACM Trans. Graph. 30, 2, Article 11 (April 2011), 14 pages. Google ScholarDigital Library
54. Yunhao Tang, Shipra Agrawal, and Yuri Faenza. 2019. Reinforcement learning for integer programming: Learning to cut. arXiv preprint arXiv:1906.04859 (2019).Google Scholar
55. Yonglong Tian, Andrew Luo, Xingyuan Sun, Kevin Ellis, William T. Freeman, Joshua B. Tenenbaum, and Jiajun Wu. 2019. Learning to Infer and Execute 3D Shape Programs. In International Conference on Learning Representations.Google Scholar
56. Carlos A. Vanegas, Ignacio Garcia-Dorado, Daniel G. Aliaga, Bedrich Benes, and Paul Waddell. 2012. Inverse Design of Urban Procedural Models. ACM Trans. Graph. 31, 6, Article 168 (Nov. 2012), 11 pages. Google ScholarDigital Library
57. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30. Curran Associates, Inc., 5998–6008. https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdfGoogle ScholarDigital Library
58. Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017. Graph attention networks. arXiv preprint arXiv:1710.10903 (2017).Google Scholar
59. K.J. Weiler. 1986. Topological structures for geometric modeling. University Microfilms.Google Scholar
60. Daniel Weiss. 2009. Geometry-based structural optimization on CAD specification trees. Ph.D. Dissertation. ETH Zurich.Google Scholar
61. Rundi Wu, Yixin Zhuang, Kai Xu, Hao Zhang, and Baoquan Chen. 2020. Pq-net: A generative part seq2seq network for 3d shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 829–838.Google ScholarCross Ref
62. Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 2015. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1912–1920.Google Scholar
63. Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. 2018. How powerful are graph neural networks? arXiv preprint arXiv:1810.00826 (2018).Google Scholar
64. Zhibo Zhang, Prakhar Jaiswal, and Rahul Rai. 2018. FeatureNet: machining feature recognition based on 3D convolution neural network. Computer-Aided Design 101 (2018), 12–22.Google ScholarCross Ref
65. Qingnan Zhou and Alec Jacobson. 2016. Thingi10K: A Dataset of 10,000 3D-Printing Models. arXiv preprint arXiv:1605.04797 (2016).Google Scholar
66. Chuhang Zou, Ersin Yumer, Jimei Yang, Duygu Ceylan, and Derek Hoiem. 2017. 3d-prnn: Generating shape primitives with recurrent neural networks. In Proceedings of the IEEE International Conference on Computer Vision. 900–909.Google ScholarCross Ref