“Free2CAD: parsing freehand drawings into CAD commands” by Li, Pan, Bousseau and Mitra

CAD modeling, despite being the industry-standard, remains restricted to usage by skilled practitioners due to two key barriers. First, the user must be able to mentally parse a final shape into a valid sequence of supported CAD commands; and second, the user must be sufficiently conversant with CAD software packages to be able to execute the corresponding CAD commands. As a step towards addressing both these challenges, we present Free2CAD wherein the user can simply sketch the final shape and our system parses the input strokes into a sequence of commands expressed in a simplified CAD language. When executed, these commands reproduce the sketched object. Technically, we cast sketch-based CAD modeling as a sequence-to-sequence translation problem, for which we leverage the powerful Transformers neural network architecture. Given the sequence of pen strokes as input, we introduce the new task of grouping strokes that correspond to individual CAD operations. We combine stroke grouping with geometric fitting of the operation parameters, such that intermediate groups are geometrically corrected before being reused, as context, for subsequent steps in the sequence inference. Although trained on synthetically-generated data, we demonstrate that Free2CAD generalizes to sketches created from real-world CAD models as well as to sketches drawn by novice users.Code and data are at https://github.com/Enigma-li/Free2CAD.

1. Autodesk. 2019. TinkerCAD. https://www.tinkercad.com/Google Scholar
2. Seok-Hyung Bae, Ravin Balakrishnan, and Karan Singh. 2008. ILoveSketch: as-natural-as-possible sketching system for creating 3d curve models. In Proc. UIST. ACM, 151–160.Google ScholarDigital Library
3. Suresh K. Bhavnani, Bonnie E. John, and Ulrich Flemming. 1999. The Strategic Use of CAD: An Empirically Inspired, Theory-Based Course. In Proc. of the ACM SIGCHI Conference on Human Factors in Computing Systems. 183–190.Google Scholar
4. Alexandra Bonnici, Alican Akman, Gabriel Calleja, Kenneth P Camilleri, Patrick Fehling, Alfredo Ferreira, Florian Hermuth, Johann Habakuk Israel, Tom Landwehr, Juncheng Liu, et al. 2019. Sketch-based interaction and modeling: where do we stand? Artificial intelligence for engineering design analysis and manufacturing (2019), 1–19.Google Scholar
5. Yu Chen, Jianzhuang Liu, and Xiaoou Tang. 2007. A Divide-and-Conquer Approach to 3D Object Reconstruction from Line Drawings. In ICCV.Google Scholar
6. Ivan Chester. 2006. Teaching for CAD expertise. International Journal of Technology and Design Education 17 (2006), 23–35.Google ScholarCross Ref
7. Bert Dodson. 1990. Keys to drawing. Penguin.Google Scholar
8. Tao Du, Jeevana Priya Inala, Yewen Pu, Andrew Spielberg, Adriana Schulz, Daniela Rus, Armando Solar-Lezama, and Wojciech Matusik. 2018. InverseCSG: Automatic Conversion of 3D Models to CSG Trees. ACM Transactions on Graphics (Proc. SIGGRAPH Asia 37, 6 (2018).Google Scholar
9. Betty Edwards. 1979. Drawing on the Right Side of the Brain. Penguin.Google Scholar
10. Kevin Ellis, Maxwell Nye, Yewen Pu, Felix Sosa, Josh Tenenbaum, and Armando Solar-Lezama. 2019. Write, Execute, Assess: Program Synthesis with a REPL. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 32.Google Scholar
11. Kevin Ellis, Daniel Ritchie, Armando Solar-Lezama, and Josh Tenenbaum. 2018. Learning to infer graphics programs from hand-drawn images. In Advances in neural information processing systems. 6059–6068.Google Scholar
12. Yaroslav Ganin, Sergey Bartunov, Yujia Li, Ethan Keller, and Stefano Saliceti. 2021. Computer-aided design as language. Advances in Neural Information Processing Systems 34 (2021).Google Scholar
13. Yotam Gingold, Takeo Igarashi, and Denis Zorin. 2009. Structured Annotations for 2D-to-3D Modeling. ACM Transactions on Graphics (Proc. SIGGRAPH Asia) 28, 5 (2009).Google Scholar
14. Yulia Gryaditskaya, Felix Hähnlein, Chenxi Liu, Alla Sheffer, and Adrien Bousseau. 2020. Lifting Freehand Concept Sketches into 3D. ACM Transactions on Graphics (2020).Google Scholar
15. Yulia Gryaditskaya, Mark Sypesteyn, Jan Willem Hoftijzer, Sylvia Pont, Fredo Durand, and Adrien Bousseau. 2019. OpenSketch: A Richly-Annotated Dataset of Product Design Sketches. ACM Transactions on Graphics (Proc. SIGGRAPH Asia) (2019).Google Scholar
16. David Ha and Douglas Eck. 2018. A Neural Representation of Sketch Drawings. In International Conference on Learning Representations (ICLR).Google Scholar
17. Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. 2021. Masked Autoencoders Are Scalable Vision Learners. CoRR abs/2111.06377 (2021).Google Scholar
18. Brian Hempel, Justin Lubin, and Ravi Chugh. 2019. Sketch-n-sketch: Output-directed programming for SVG. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology. 281–292.Google ScholarDigital Library
19. Haibin Huang, Evangelos Kalogerakis, Ersin Yumer, and Radomir Mech. 2016. Shape Synthesis from Sketches via Procedural Models and Convolutional Networks. IEEE Transactions on Visualization and Computer Graphics (TVCG) 22, 10 (2016), 1.Google ScholarDigital Library
20. R. Kenny Jones, Theresa Barton, Xianghao Xu, Kai Wang, Ellen Jiang, Paul Guerrero, Niloy J. Mitra, and Daniel Ritchie. 2020. ShapeAssembly: Learning to Generate Programs for 3D Shape Structure Synthesis. ACM Transactions on Graphics (Proc. Siggraph Asia) 39, 6 (2020), Article 234.Google ScholarDigital Library
21. Joaquim A Jorge, Nelson F Silva, and Tiago D Cardoso. 2003. GIDeS++: A Rapid Prototyping Tool for Mould Design. In Proceedings of the Rapid Product Development Event RDP.Google Scholar
22. Kacper Kania, Maciej Zieba, and Tomasz Kajdanowicz. 2020. UCSG-NET- Unsupervised Discovering of Constructive Solid Geometry Tree. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 33.Google Scholar
23. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations (ICLR).Google Scholar
24. Sebastian Koch, Albert Matveev, Zhongshi Jiang, Francis Williams, Alexey Artemov, Evgeny Burnaev, Marc Alexa, Denis Zorin, and Daniele Panozzo. 2019. ABC: A Big CAD Model Dataset For Geometric Deep Learning. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
25. Changjian Li, Hao Pan, Adrien Bousseau, and Niloy J Mitra. 2020. Sketch2cad: Sequential cad modeling by sketching in context. ACM Transactions on Graphics (Proc. SIGGRAPH Asia) 39, 6 (2020), 1–14.Google Scholar
26. Changjian Li, Hao Pan, Yang Liu, Alla Sheffer, and Wenping Wang. 2017. BendSketch: Modeling Freeform Surfaces Through 2D Sketching. ACM Trans. Graph. (Proc. SIGGRAPH) 36, 4 (2017).Google ScholarDigital Library
27. Ke Li, Kaiyue Pang, Jifei Song, Yi-Zhe Song, Tao Xiang, Timothy M Hospedales, and Honggang Zhang. 2018. Universal sketch perceptual grouping. In Proceedings of the european conference on computer vision (ECCV). 582–597.Google ScholarDigital Library
28. Lei Li, Hongbo Fu, and Chiew-Lan Tai. 2019. Fast Sketch Segmentation and Labeling With Deep Learning. IEEE Computer Graphics and Applications 39, 2 (2019), 38–51.Google ScholarCross Ref
29. H Lipson and M Shpitalni. 1996. Optimization-based reconstruction of a 3D object from a single freehand line drawing. Computer-Aided Design 28, 8 (1996), 651–663.Google ScholarCross Ref
30. Tsvetomila Mihaylova and André FT Martins. 2019. Scheduled sampling for transformers. arXiv preprint arXiv:1906.07651 (2019).Google Scholar
31. Chandrakana Nandi, James R. Wilcox, Pavel Panchekha, Taylor Blau, Dan Grossman, and Zachary Tatlock. 2018. Functional Programming for Compiling and Decompiling Computer-aided Design. Proceedings of the ACM on Programming Languages 2, ICFP (2018), 99:1–99:31.Google ScholarDigital Library
32. Gen Nishida, Ignacio Garcia-Dorado, Daniel G. Aliaga, Bedrich Benes, and Adrien Bousseau. 2016. Interactive Sketching of Urban Procedural Models. ACM Transactions on Graphics (SIGGRAPH Conference Proceedings) (2016).Google Scholar
33. GüNay Orbay and Levent Burak Kara. 2012. Sketch-based surface design using malleable curve networks. Comput. Graph. Forum 36, 8 (2012).Google Scholar
34. Wamiq Reyaz Para, Shariq Farooq Bhat, Paul Guerrero, Tom Kelly, Niloy Mitra, Leonidas Guibas, and Peter Wonka. 2021. SketchGen: Generating Constrained CAD Sketches. arXiv:2106.02711 [cs.LG]Google Scholar
35. Yonggang Qi and Zheng-Hua Tan. 2019. SketchSegNet+: An End-to-End Learning of RNN for Multi-Class Sketch Semantic Segmentation. IEEE Access 7 (2019), 102717–102726.Google ScholarCross Ref
36. Leo Sampaio Ferraz Ribeiro, Tu Bui, John Collomosse, and Moacir Ponti. 2020. Sketch-former: Transformer-based representation for sketched structure. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
37. Ryan Schmidt, Azam Khan, Karan Singh, and Gord Kurtenbach. 2009. Analytic drawing of 3D scaffolds. In ACM transactions on graphics (TOG), Vol. 28. 149.Google Scholar
38. Rosália G. Schneider and Tinne Tuytelaars. 2016. Example-Based Sketch Segmentation and Labeling Using CRFs. ACM Trans. Graph. 35, 5, Article 151 (July 2016), 9 pages.Google ScholarDigital Library
39. Ari Seff, Wenda Zhou, Nick Richardson, and Ryan P Adams. 2021. Vitruvion: A Generative Model of Parametric CAD Sketches. arXiv preprint arXiv:2109.14124 (2021).Google Scholar
40. Tianjia Shao, Wilmot Li, Kun Zhou, Weiwei Xu, Baining Guo, and Niloy J. Mitra. 2013. Interpreting Concept Sketches. ACM Transactions on Graphics (Proc. SIGGRAPH) 32, 4 (2013), 10 pages.Google Scholar
41. Shapr3D. 2016. Shapr3D. https://www.shapr3d.com/Google Scholar
42. Gopal Sharma, Rishabh Goyal, Difan Liu, Evangelos Kalogerakis, and Subhransu Maji. 2018. CSGNet: Neural Shape Parser for Constructive Solid Geometry. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
43. Alex Shtof, Alexander Agathos, Yotam Gingold, Ariel Shamir, and Daniel Cohen-Or. 2013. Geosemantic snapping for sketch-based modeling. In Computer graphics forum, Vol. 32. Wiley Online Library, 245–253.Google Scholar
44. Roselien Steur. 2011. Sketching: The Basics. BIS.Google Scholar
45. Yonglong Tian, Andrew Luo, Xingyuan Sun, Kevin Ellis, William T. Freeman, Joshua B. Tenenbaum, and Jiajun Wu. 2019. Learning to Infer and Execute 3D Shape Programs. In International Conference on Learning Representations.Google Scholar
46. Trimble. 2019. SketchUp. https://www.sketchup.com/Google Scholar
47. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems, Vol. 30.Google Scholar
48. Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer Networks. In Advances in Neural Information Processing Systems, Vol. 28. Curran Associates, Inc.Google Scholar
49. Zeyu Wang, Sherry Qiu, Nicole Feng, Holly Rushmeier, Leonard McMillan, and Julie Dorsey. 2021. Tracing versus freehand for evaluating computer-generated drawings. ACM Transactions on Graphics (TOG) 40, 4 (2021), 1–12.Google ScholarDigital Library
50. Karl D. D. Willis, Yewen Pu, Jieliang Luo, Hang Chu, Tao Du, Joseph G. Lambourne, Armando Solar-Lezama, and Wojciech Matusik. 2021. Fusion 360 Gallery: A Dataset and Environment for Programmatic CAD Construction from Human Design Sequences. ACM Transactions on Graphics (TOG) 40, 4 (2021).Google ScholarDigital Library
51. Q. Wu, K. Xu, and J. Wang. 2018. Constructing 3D CSG Models from 3D Raw Point Clouds. Computer Graphics Forum 37, 5 (2018).Google Scholar
52. Rundi Wu, Chang Xiao, and Changxi Zheng. 2021. DeepCAD: A Deep Generative Network for Computer-Aided Design Models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 6772–6782.Google ScholarCross Ref
53. Baoxuan Xu, William Chang, Alla Sheffer, Adrien Bousseau, James McCrae, and Karan Singh. 2014. True2Form: 3D curve networks from 2D sketches via selective regularization. ACM Transactions on Graphics 33, 4 (2014).Google ScholarDigital Library
54. Xianghao Xu, Wenzhe Peng, Chin-Yi Cheng, Karl D. D. Willis, and Daniel Ritchie. 2021. Inferring CAD Modeling Sequences Using Zone Graphs. In CVPR.Google Scholar
55. Lumin Yang, Jiajie Zhuang, Hongbo Fu, Xiangzhi Wei, Kun Zhou, and Youyi Zheng. 2021. SketchGNN: Semantic Sketch Segmentation with Graph Neural Networks. ACM Transactions on Graphics 40, 3, Article 28 (Aug. 2021), 13 pages.Google ScholarDigital Library
56. Fenggen Yu, Zhiqin Chen, Manyi Li, Aditya Sanghi, Hooman Shayani, Ali Mahdavi-Amiri, and Hao Zhang. 2021. CAPRI-Net: Learning Compact CAD Shapes with Adaptive Primitive Assembly. arXiv:2104.05652 [cs.CV]Google Scholar
57. Robert C. Zeleznik, Kenneth P. Herndon, and John F. Hughes. 1996. SKETCH: An Interface for Sketching 3D Scenes. In Proceedings of SIGGRAPH (Computer Graphics Proceedings, Annual Conference Series). 163–170.Google Scholar