“OpenSketch: a richly-annotated dataset of product design sketches” by Gryaditskaya, Sypesteyn, Hoftijzer, Pont, Durand, et al. …
Conference:
Type(s):
Title:
- OpenSketch: a richly-annotated dataset of product design sketches
Session/Category Title: Hairy & Sketchy Geometry
Presenter(s)/Author(s):
Moderator(s):
Abstract:
Product designers extensively use sketches to create and communicate 3D shapes and thus form an ideal audience for sketch-based modeling, non-photorealistic rendering and sketch filtering. However, sketching requires significant expertise and time, making design sketches a scarce resource for the research community. We introduce OpenSketch, a dataset of product design sketches aimed at offering a rich source of information for a variety of computer-aided design tasks. OpenSketch contains more than 400 sketches representing 12 man-made objects drawn by 7 to 15 product designers of varying expertise. We provided participants with front, side and top views of these objects, and instructed them to draw from two novel perspective viewpoints. This drawing task forces designers to construct the shape from their mental vision rather than directly copy what they see. They achieve this task by employing a variety of sketching techniques and methods not observed in prior datasets. Together with industrial design teachers, we distilled a taxonomy of line types and used it to label each stroke of the 214 sketches drawn from one of the two viewpoints. While some of these lines have long been known in computer graphics, others remain to be reproduced algorithmically or exploited for shape inference. In addition, we also asked participants to produce clean presentation drawings from each of their sketches, resulting in aligned pairs of drawings of different styles. Finally, we registered each sketch to its reference 3D model by annotating sparse correspondences. We provide an analysis of our annotated sketches, which reveals systematic drawing strategies over time and shapes, as well as a positive correlation between presence of construction lines and accuracy. Our sketches, in combination with provided annotations, form challenging benchmarks for existing algorithms as well as a great source of inspiration for future developments. We illustrate the versatility of our data by using it to test a 3D reconstruction deep network trained on synthetic drawings, as well as to train a filtering network to convert concept sketches into presentation drawings. We distribute our dataset under the Creative Commons CC0 license: https://ns.inria.fr/d3/OpenSketch.
References:
1. Kfir Aberman, Jing Liao, Mingyi Shi, Dani Lischinski, Baoquan Chen, and Daniel Cohen-Or. 2018. Neural Best-Buddies: Sparse Cross-Domain Correspondence. ACM Transactions on Graphics (Proc. SIGGRAPH) 37, 4 (2018), 69.Google ScholarDigital Library
2. Martín Arjovsky, Soumith Chintala, and Léon Bottou. 2017. Wasserstein Generative Adversarial Networks. In International Conference on Machine Learning (ICML).Google Scholar
3. Seok-Hyung Bae, Ravin Balakrishnan, and Karan Singh. 2008. ILoveSketch: as-natural-as-possible sketching system for creating 3d curve models. In ACM Symposium on User Interface Software and Technology (UIST). ACM.Google ScholarDigital Library
4. Seok-Hyung Bae, Ravin Balakrishnan, and Karan Singh. 2009. EverybodyLovesSketch: 3D sketching for a broader audience. In ACM Symposium on User Interface Software and Technology (UIST). ACM.Google ScholarDigital Library
5. Itamar Berger, Ariel Shamir, Moshe Mahler, Elizabeth Carter, and Jessica Hodgins. 2013. Style and Abstraction in Portrait Sketching. ACM Transactions on Graphics (Proc. SIGGRAPH) 32, 4 (2013).Google Scholar
6. Mikhail Bessmeltsev and Justin Solomon. 2018. Vectorization of Line Drawings via PolyVector Fields. ACM Transactions on Graphics 38, 1 (2018).Google Scholar
7. Forrester Cole, Aleksey Golovinskiy, Alex Limpaecher, Heather Stoddart Barros, Adam Finkelstein, Thomas Funkhouser, and Szymon Rusinkiewicz. 2008. Where do people draw lines? ACM Transactions on Graphics (Proc. SIGGRAPH) 27, 3 (2008).Google Scholar
8. Chris De Paoli and Karan Singh. 2015. SecondSkin: sketch-based construction of layered 3D models. ACM Transactions on Graphics (Proc. SIGGRAPH) 34, 4 (2015).Google ScholarDigital Library
9. Johanna Delanoy, Mathieu Aubry, Phillip Isola, Alexei A Efros, and Adrien Bousseau. 2018. 3D Sketching using Multi-View Deep Volumetric Prediction. Proceedings of the ACM on Computer Graphics and Interactive Techniques 1, 1 (2018), 21.Google ScholarDigital Library
10. Claudia Eckert, Alan Blackwell, Martin Stacey, Christopher Earl, and Luke Church. 2012. Sketching across design domains: Roles and formalities. Artificial Intelligence for Engineering Design, Analysis and Manufacturing 26 (8 2012), 245–266.Google Scholar
11. Koos Eissen and Roselien Steur. 2008. Sketching: Drawing Techniques for Product Designers. Bis Publishers.Google Scholar
12. Koos Eissen and Roselien Steur. 2011. Sketching: The Basics. Bis Publishers.Google Scholar
13. Mathias Eitz, James Hays, and Marc Alexa. 2012. How Do Humans Sketch Objects? ACM Transactions on Graphics (Proc. SIGGRAPH) 31, 4 (2012).Google Scholar
14. Jean-Dominique Favreau, Florent Lafarge, and Adrien Bousseau. 2016. Fidelity vs. Simplicity: a Global Approach to Line Drawing Vectorization. ACM Transactions on Graphics (Proc. SIGGRAPH) (2016).Google ScholarDigital Library
15. Yotam Gingold, Takeo Igarashi, and Denis Zorin. 2009. Structured annotations for 2D-to-3D modeling. ACM Transactions on Graphics (Proc. SIGGRAPH Asia) 28, 5 (2009).Google Scholar
16. Gabriela Goldschmidt. 1991. The Dialectics of Sketching. Creativity Research Journal 4, 2 (1991).Google ScholarCross Ref
17. Richard Hartley and Andrew Zisserman. 2000. Multiple View Geometry in Computer Vision Second Edition. Cambridge University Press (2000).Google ScholarDigital Library
18. James W. Hennessey, Han Liu, Holger Winnemöller, Mira Dontcheva, and Niloy J. Mitra. 2017. How2Sketch: Generating Easy-to-follow Tutorials for Sketching 3D Objects. ACM Symposium on Interactive 3D Graphics and Games (I3D) (2017).Google ScholarDigital Library
19. Aaron Hertzmann and Denis Zorin. 2000. Illustrating smooth surfaces. ACM SIGGRAPH (2000).Google Scholar
20. Jan Willem Hoftijzer. 2018. You can’t see it if you don’t draw it. In Design para acessibilidade e inclusão. Blucher.Google Scholar
21. Jan Willem Hoftijzer, Mark Sypesteyn, Rik de Reuver, and Jort Nijhuis. 2018. A Typology of Design Sketches, Defined by Communication Factors. In Proc. International Conference on Engineering and Product Design Education.Google Scholar
22. Emmanuel Iarussi, David Bommes, and Adrien Bousseau. 2015. BendFields: Regularized Curvature Fields from Rough Concept Sketches. ACM Transactions on Graphics 34, 3 (2015).Google ScholarDigital Library
23. Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017. Image-to-Image Translation with Conditional Adversarial Networks. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017).Google Scholar
24. Swarna Keshavabhotla, Blake Williford, Shalini Kumar, Ethan Hilton, Paul Taele, Wayne Li, Julie Linsey, and Tracy Hammond. 2017. Conquering the Cube: Learning to Sketch Primitives in Perspective with an Intelligent Tutoring System. In Sketch-Based Interfaces and Modeling (SBIM).Google Scholar
25. Byungsoo Kim, Oliver Wang, A. Cengiz Öztireli, and Markus Gross. 2018. Semantic Segmentation for Line Drawing Vectorization Using Neural Networks. Computer Graphics Forum (Proc. Eurographics) 37, 2 (2018), 329–338.Google ScholarCross Ref
26. Yongkwan Kim and Seok-Hyung Bae. 2016. SketchingWithHands: 3D Sketching Hand-held Products with First-Person Hand Posture. In ACM Symposium on User Interface Software and Technology (UIST). ACM, 12.Google ScholarDigital Library
27. Jan J. Koenderink, Andrea J. Van Doorn, and Astrid M. L. Kappers. 1992. Surface perception in pictures. Perception & Psychophysics 52, 5 (1992), 487–496.Google ScholarCross Ref
28. Changjian Li, Hao Pan, Yang Liu, Alla Sheffer, and Wenping Wang. 2017. BendSketch: Modeling Freeform Surfaces Through 2D Sketching. ACM Transactions on Graphics (Proc. SIGGRAPH) 36, 4 (2017).Google ScholarDigital Library
29. Changjian Li, Hao Pan, Yang Liu, Xin Tong, Alla Sheffer, and Wenping Wang. 2018. Robust flow-guided neural prediction for sketch-based freeform surface modeling. ACM Transaction on Graphics (Proc. SIGGRAPH Asia) (2018), 238.Google Scholar
30. Alex Limpaecher, Nicolas Feltman, Adrien Treuille, and Michael Cohen. 2013. Real-time Drawing Assistance Through Crowdsourcing. ACM Transactions on Graphics (Proc. SIGGRAPH) 32, 4 (2013).Google Scholar
31. Chenxi Liu, Enrique Rosales, and Alla Sheffer. 2018. StrokeAggregator: Consolidating Raw Sketches into Artist-Intended Curve Drawings. ACM Transactions on Graphics (Proc. SIGGRAPH) 37, 4 (2018).Google ScholarDigital Library
32. Xueting Liu, Tien-Tsin Wong, and Pheng-Ann Heng. 2015. Closure-aware Sketch Simplification. ACM Transactions on Graphics (Proc. SIGGRAPH Asia) 34, 6 (2015).Google Scholar
33. Zhaoliang Lun, Matheus Gadelha, Evangelos Kalogerakis, Subhransu Maji, and Rui Wang. 2017. 3D shape reconstruction from sketches via multi-view convolutional networks. In IEEE International Conference on 3D Vision (3DV). 67–77.Google ScholarCross Ref
34. Yutaka Ohtake, Alexander Belyaev, and Hans-Peter Seidel. 2004. Ridge-valley Lines on Meshes via Implicit Surface Fitting. ACM Transaction on Graphics (Proc. SIGGRAPH) 23, 3 (2004).Google Scholar
35. Gunay Orbay and Levent Burak Kara. 2011. Beautification of Design Sketches Using Trainable Stroke Clustering and Curve Fitting. IEEE Transactions on Visualization and Computer Graphics (TVCG) 17, 5 (2011).Google ScholarDigital Library
36. GüNay Orbay and Levent Burak Kara. 2012. Sketch-based surface design using malleable curve networks. Computers & Graphics 36, 8 (2012).Google Scholar
37. Hao Pan, Yang Liu, Alla Sheffer, Nicholas Vining, Chang-Jian Li, and Wenping Wang. 2015. Flow aligned surfacing of curve networks. ACM Transactions on Graphics (Proc. SIGGRAPH) 34, 4 (2015), 127.Google ScholarDigital Library
38. Eujin Pei, Ian Campbell, and Mark Evans. 2011. A Taxonomic Classification of Visual Design Representations Used by Industrial Designers and Engineering Designers. The Design Journal 14, 1 (2011), 64–91.Google ScholarCross Ref
39. Alan Pipes. 2007. Drawing for Designers. Laurence King.Google Scholar
40. D. Powell. 1986. Design rendering techniques: a guide to drawing and presenting design ideas. North Light.Google Scholar
41. A.T. Purcell and J.S. Gero. 1998. Drawings and the design process: A review of protocol studies in design and other disciplines and related research in cognitive psychology. Design Studies 19, 4 (1998).Google Scholar
42. Scott Robertson and Thomas Bertling. 2013. How to Draw: drawing and sketching objects and environments from your imagination. Design Studio Press.Google Scholar
43. Patsorn Sangkloy, Nathan Burnell, Cusuh Ham, and James Hays. 2016. The Sketchy Database: Learning to Retrieve Badly Drawn Bunnies. ACM Transactions on Graphics (Proc. SIGGRAPH) (2016).Google ScholarDigital Library
44. Ryan Schmidt, Azam Khan, Gord Kurtenbach, and Karan Singh. 2009a. On Expert Performance in 3D Curve-drawing Tasks. In Sketch-Based Interfaces and Modeling (SBIM). 8.Google Scholar
45. Ryan Schmidt, Azam Khan, Karan Singh, and Gord Kurtenbach. 2009b. Analytic Drawing of 3D Scaffolds. ACM Transactions on Graphics (Proc. SIGGRAPH Asia) (2009).Google Scholar
46. Cloud Shao, Adrien Bousseau, Alla Sheffer, and Karan Singh. 2012. CrossShade: shading concept sketches using cross-section curves. ACM Transactions on Graphics (Proc. SIGGRAPH) 31, 4 (2012).Google ScholarDigital Library
47. Edgar Simo-Serra, Satoshi Iizuka, and Hiroshi Ishikawa. 2018. Mastering Sketching: Adversarial Augmentation for Structured Prediction. ACM Transactions on Graphics 37, 1 (2018).Google ScholarDigital Library
48. Wanchao Su, Dong Du, Xin Yang, Shizhe Zhou, and Hongbo Fu. 2018. Interactive Sketch-Based Normal Map Generation with Deep Neural Networks. Proceedings of the ACM on Computer Graphics and Interactive Techniques 1, 1 (2018).Google ScholarDigital Library
49. John Tchalenko. 2009. Segmentation and accuracy in copying and drawing: Experts and beginners. Vision research 49, 8 (2009).Google Scholar
50. Baoxuan Xu, William Chang, Alla Sheffer, Adrien Bousseau, James McCrae, and Karan Singh. 2014. True2Form: 3D curve networks from 2D sketches via selective regularization. ACM Transactions on Graphics (Proc. SIGGRAPH) 33, 4 (2014).Google ScholarDigital Library
51. Qiuying Xu, Yotam Gingold, and Karan Singh. 2015. Inverse toon shading: interactive normal field modeling with isophotes. In Sketch-Based Interfaces and Modeling (SBIM).Google Scholar
52. Changqing Zou, Qian Yu, Ruofei Du, Haoran Mo, Yi-Zhe Song, Tao Xiang, Chengying Gao, Baoquan Chen, and Hao Zhang. 2018. SketchyScene: Richly-Annotated Scene Sketches. In European Conference on Computer Vison (ECCV).Google Scholar
