“Sketch2CAD: sequential CAD modeling by sketching in context” by Li, Pan, Bousseau and Mitra
Conference:
Type(s):
Title:
- Sketch2CAD: sequential CAD modeling by sketching in context
Session/Category Title: All About Sketches
Presenter(s)/Author(s):
Abstract:
We present a sketch-based CAD modeling system, where users create objects incrementally by sketching the desired shape edits, which our system automatically translates to CAD operations. Our approach is motivated by the close similarities between the steps industrial designers follow to draw 3D shapes, and the operations CAD modeling systems offer to create similar shapes. To overcome the strong ambiguity with parsing 2D sketches, we observe that in a sketching sequence, each step makes sense and can be interpreted in the context of what has been drawn before. In our system, this context corresponds to a partial CAD model, inferred in the previous steps, which we feed along with the input sketch to a deep neural network in charge of interpreting how the model should be modified by that sketch. Our deep network architecture then recognizes the intended CAD operation and segments the sketch accordingly, such that a subsequent optimization estimates the parameters of the operation that best fit the segmented sketch strokes. Since there exists no datasets of paired sketching and CAD modeling sequences, we train our system by generating synthetic sequences of CAD operations that we render as line drawings. We present a proof of concept realization of our algorithm supporting four frequently used CAD operations. Using our system, participants are able to quickly model a large and diverse set of objects, demonstrating Sketch2CAD to be an alternate way of interacting with current CAD modeling systems.
References:
1. Autodesk. 2019a. 3ds Max. https://www.autodesk.com/products/3ds-max/overviewGoogle Scholar
2. Autodesk. 2019b. Maya. https://www.autodesk.com/products/maya/overviewGoogle Scholar
3. Autodesk. 2019. TinkerCAD. https://www.tinkercad.com/Google Scholar
4. Armen Avetisyan, Manuel Dahnert, Angela Dai, Manolis Savva, Angel X. Chang, and Matthias Nießner. 2019. Scan2CAD: Learning CAD Model Alignment in RGB-D Scans. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google ScholarCross Ref
5. Seok-Hyung Bae, Ravin Balakrishnan, and Karan Singh. 2008. ILoveSketch: as-natural-as-possible sketching system for creating 3d curve models. In Proc. ACM UIST. ACM, 151–160.Google ScholarDigital Library
6. Alexandra Bonnici, Alican Akman, Gabriel Calleja, Kenneth P Camilleri, Patrick Fehling, Alfredo Ferreira, Florian Hermuth, Johann Habakuk Israel, Tom Landwehr, Juncheng Liu, et al. 2019. Sketch-based interaction and modeling: where do we stand? AI EDAM (2019), 1–19.Google Scholar
7. Frederic Cordier, Hyewon Seo, Mahmoud Melkemi, and Nickolas S. Sapidis. 2013. Inferring Mirror Symmetric 3D Shapes from Sketches. Computer Aided Design 45, 2 (Feb. 2013), 301–311.Google ScholarDigital Library
8. Chris De Paoli and Karan Singh. 2015. SecondSkin: Sketch-Based Construction of Layered 3D Models. ACM Transactions on Graphics (Proc. SIGGRAPH) 34, 4, Article 126 (July 2015), 10 pages.Google Scholar
9. Johanna Delanoy, Mathieu Aubry, Phillip Isola, Alexei A Efros, and Adrien Bousseau. 2018. 3D Sketching using Multi-View Deep Volumetric Prediction. Proceedings of the ACM on Computer Graphics and Interactive Techniques 1, 1 (2018), 21.Google ScholarDigital Library
10. Jonathan D. Denning, William B. Kerr, and Fabio Pellacini. 2011. MeshFlow: Interactive Visualization of Mesh Construction Sequences. ACM Trans. Graph. 30, 4, Article Article 66 (July 2011), 8 pages.Google ScholarDigital Library
11. Tao Du, Jeevana Priya Inala, Yewen Pu, Andrew Spielberg, Adriana Schulz, Daniela Rus, Armando Solar-Lezama, and Wojciech Matusik. 2018. InverseCSG: Automatic Conversion of 3D Models to CSG Trees. ACM Transactions on Graphics (Proc. SIGGRAPH Asia 37, 6 (2018).Google Scholar
12. Koos Eissen and Roselien Steur. 2008. Sketching: Drawing Techniques for Product Designers. Bis Publishers.Google Scholar
13. K. Eissen and R. Steur. 2011. Sketching: The Basics. BIS.Google Scholar
14. Kevin Ellis, Daniel Ritchie, Armando Solar-Lezama, and Josh Tenenbaum. 2018. Learning to infer graphics programs from hand-drawn images. In Advances in neural information processing systems. 6059–6068.Google Scholar
15. Jean-Dominique Favreau, Florent Lafarge, and Adrien Bousseau. 2015. Line Drawing Interpretation in a Multi-View Context. In Proceedings of the Conference on Computer Vision and Pattern Recognition. IEEE.Google ScholarCross Ref
16. Yotam Gingold, Takeo Igarashi, and Denis Zorin. 2009. Structured Annotations for 2D-to-3D Modeling. ACM Transactions on Graphics (Proc. SIGGRAPH Asia) 28, 5 (2009).Google Scholar
17. Yulia Gryaditskaya, Mark Sypesteyn, Jan Willem Hoftijzer, Sylvia Pont, Frédo Durand, and Adrien Bousseau. 2019. OpenSketch: A Richly-Annotated Dataset of Product Design Sketches. ACM Trans. Graph. (SIGGRAPH Asia) 38, 6 (November 2019).Google Scholar
18. Haibin Huang, Evangelos Kalogerakis, Ersin Yumer, and Radomir Mech. 2016. Shape Synthesis from Sketches via Procedural Models and Convolutional Networks. IEEE Transactions on Visualization and Computer Graphics (TVCG) 22, 10 (2016), 1.Google ScholarDigital Library
19. Takeo Igarashi, Satoshi Matsuoka, and Hidehiko Tanaka. 1999. Teddy: A Sketching Interface for 3D Freeform Design. SIGGRAPH (1999).Google ScholarDigital Library
20. Joaquim A Jorge, Nelson F Silva, and Tiago D Cardoso. 2003. GIDeS++: A Rapid Prototyping Tool for Mould Design. In Proceedings of the Rapid Product Development Event RDP.Google Scholar
21. Manfred Lau, Greg Saul, Jun Mitani, and Takeo Igarashi. 2010. Modeling-in-context: user design of complementary objects with a single photo. In Proc. Sketch-Based Interfaces and Modeling.Google Scholar
22. Changjian Li, Hao Pan, Yang Liu, Xin Tong, Alla Sheffer, and Wenping Wang. 2018. Robust flow-guided neural prediction for sketch-based freeform surface modeling. ACM Transaction on Graphics (Proc. SIGGRAPH Asia) (2018), 238.Google Scholar
23. Yuwei Li, Xi Luo, Youyi Zheng, Pengfei Xu, and Hongbo Fu. 2017. SweepCanvas: Sketch-Based 3D Prototyping on an RGB-D Image. In Proc. ACM Symposium on User Interface Software and Technology (UIST) (UIST ’17).Google ScholarDigital Library
24. H Lipson and M Shpitalni. 1996. Optimization-based reconstruction of a 3D object from a single freehand line drawing. Computer-Aided Design 28, 8 (1996), 651–663.Google ScholarCross Ref
25. Zhaoliang Lun, Matheus Gadelha, Evangelos Kalogerakis, Subhransu Maji, and Rui Wang. 2017. 3D shape reconstruction from sketches via multi-view convolutional networks. In IEEE International Conference on 3D Vision (3DV). 67–77.Google ScholarCross Ref
26. Kaichun Mo, Shilin Zhu, Angel X. Chang, Li Yi, Subarna Tripathi, Leonidas J. Guibas, and Hao Su. 2019. PartNet: A Large-Scale Benchmark for Fine-Grained and Hierarchical Part-Level 3D Object Understanding. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google ScholarCross Ref
27. Fernando Naya, Joaquim Jorge, Julián Conesa, Manuel Contero, and José María Gomis. 2002. Direct modeling: from sketches to 3D models. In Proceedings of the 1st Ibero-American Symposium in Computer Graphics SIACG. 109–117.Google Scholar
28. Andrew Nealen, Takeo Igarashi, Olga Sorkine, and Marc Alexa. 2007. FiberMesh: designing freeform surfaces with 3D curves. ACM Transactions on Graphics (Proc. SIGGRAPH) 26, Article 41 (2007). Issue 3.Google Scholar
29. Gen Nishida, Ignacio Garcia-Dorado, Daniel G. Aliaga, Bedrich Benes, and Adrien Bousseau. 2016. Interactive Sketching of Urban Procedural Models. ACM Trans. Graph. (SIGGRAPH) 35, 4, Article 130 (July 2016), 11 pages.Google ScholarDigital Library
30. Patrick Paczkowski, Min H. Kim, Yann Morvan, Julie Dorsey, Holly Rushmeier, and Carol O’Sullivan. 2011. Insitu: Sketching Architectural Designs in Context. ACM Transactions on Graphics 30, 6 (2011).Google ScholarDigital Library
31. Mengqi Peng, Jun Xing, and Li-Yi Wei. 2018. Autocomplete 3D sculpting. ACM Transactions on Graphics (TOG) 37, 4 (2018), 1–15.Google ScholarDigital Library
32. A. Pipes. 2007. Drawing for designers. Laurence King.Google Scholar
33. Alec Rivers, Frédo Durand, and Takeo Igarashi. 2010. 3D Modeling with Silhouettes. ACM Transactions on Graphics (Proc. SIGGRAPH) 29, 4, Article 109 (2010), 8 pages.Google Scholar
34. Robert McNeel & Associates. 2019. Rhinoceros. https://www.rhino3d.com/Google Scholar
35. Ryan Schmidt, Azam Khan, Karan Singh, and Gord Kurtenbach. 2009. Analytic drawing of 3D scaffolds. In ACM Transactions on Graphics (Proc. SIGGRAPH Asia), Vol. 28. ACM, 149.Google ScholarDigital Library
36. Gopal Sharma, Rishabh Goyal, Difan Liu, Evangelos Kalogerakis, and Subhransu Maji. 2018. CSGNet: Neural Shape Parser for Constructive Solid Geometry. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
37. Alex Shtof, Alexander Agathos, Yotam Gingold, Ariel Shamir, and Daniel Cohen-Or. 2013. Geosemantic Snapping for Sketch-Based Modeling. Computer Graphics Forum 32, 2 (2013), 245–253.Google ScholarCross Ref
38. Wanchao Su, Dong Du, Xin Yang, Shizhe Zhou, and Hongbo Fu. 2018. Interactive Sketch-Based Normal Map Generation with Deep Neural Networks. Proceedings of the ACM on Computer Graphics and Interactive Techniques 1, 1 (2018).Google ScholarDigital Library
39. The CGAL Project. 2020. CGAL User and Reference Manual (5.0.2 ed.). CGAL Editorial Board. https://doc.cgal.org/5.0.2/Manual/packages.htmlGoogle Scholar
40. Yonglong Tian, Andrew Luo, Xingyuan Sun, Kevin Ellis, William T. Freeman, Joshua B. Tenenbaum, and Jiajun Wu. 2019. Learning to Infer and Execute 3D Shape Programs. In International Conference on Learning Representations.Google Scholar
41. Trimble. 2019. SketchUp. https://www.sketchup.com/Google Scholar
42. Yingze Wang, Yu Chen, Jianzhuang Liu, and Xiaoou Tang. 2009. 3D reconstruction of curved objects from single 2D line drawings. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
43. Baoxuan Xu, William Chang, Alla Sheffer, Adrien Bousseau, James McCrae, and Karan Singh. 2014. True2Form: 3D curve networks from 2D sketches via selective regularization. ACM Transactions on Graphics (Proc. SIGGRAPH) 33, 4 (2014).Google ScholarDigital Library
44. Pengfei Xu, Hongbo Fu, Youyi Zheng, Karan Singh, Hui Huang, and Chiew-Lan Tai. 2019. Model-Guided 3D Sketching. IEEE Transactions on Visualization and Computer Graphics 25, 10 (2019), 2927–2939.Google ScholarCross Ref
45. Robert C. Zeleznik, Kenneth P. Herndon, and John F. Hughes. 1996. SKETCH: An Interface for Sketching 3D Scenes. In Proceedings of SIGGRAPH (Computer Graphics Proceedings, Annual Conference Series). 163–170.Google Scholar
46. Youyi Zheng, Han Liu, Julie Dorsey, and Niloy Mitra. 2016. SMART CANVAS : Context-inferred Interpretation of Sketches for Preparatory Design Studies. (2016).Google Scholar
