“Lifting freehand concept sketches into 3D” by Gryaditskaya, Hähnlein, Liu, Sheffer and Bousseau
Conference:
Type(s):
Title:
- Lifting freehand concept sketches into 3D
Session/Category Title: All About Sketches
Presenter(s)/Author(s):
Abstract:
We present the first algorithm capable of automatically lifting real-world, vector-format, industrial design sketches into 3D. Targeting real-world sketches raises numerous challenges due to inaccuracies, use of overdrawn strokes, and construction lines. In particular, while construction lines convey important 3D information, they add significant clutter and introduce multiple accidental 2D intersections. Our algorithm exploits the geometric cues provided by the construction lines and lifts them to 3D by computing their intended 3D intersections and depths. Once lifted to 3D, these lines provide valuable geometric constraints that we leverage to infer the 3D shape of other artist drawn strokes. The core challenge we address is inferring the 3D connectivity of construction and other lines from their 2D projections by separating 2D intersections into 3D intersections and accidental occlusions. We efficiently address this complex combinatorial problem using a dedicated search algorithm that leverages observations about designer drawing pREFERENCES, and uses those to explore only the most likely solutions of the 3D intersection detection problem. We demonstrate that our separator outputs are of comparable quality to human annotations, and that the 3D structures we recover enable a range of design editing and visualization applications, including novel view synthesis and 3D-aware scaling of the depicted shape.
References:
1. Adobe. 2020. Photoshop Perspective Warp. https://helpx.adobe.com/photoshop/using/perspective-warp.html.Google Scholar
2. Seok-Hyung Bae, Ravin Balakrishnan, and Karan Singh. 2008. ILoveSketch: as-natural-as-possible sketching system for creating 3d curve models. In Proc. UIST. ACM, 151–160.Google ScholarDigital Library
3. Mikhail Bessmeltsev and Justin Solomon. 2018. Vectorization of Line Drawings via PolyVector Fields. arXiv preprint arXiv:1801.01922 (2018).Google Scholar
4. Alexandra Bonnici, Alican Akman, Gabriel Calleja, Kenneth P Camilleri, Patrick Fehling, Alfredo Ferreira, Florian Hermuth, Johann Habakuk Israel, Tom Landwehr, Juncheng Liu, et al. 2019. Sketch-based interaction and modeling: where do we stand? Artificial intelligence for engineering design analysis and manufacturing (2019), 1–19.Google Scholar
5. Tao Chen, Zhe Zhu, Ariel Shamir, Shi-Min Hu, and Daniel Cohen-Or. 2013. 3-Sweep: Extracting Editable Objects from a Single Photo. ACM Transactions on Graphics (Proc. SIGGRAPH) 32, 6 (2013).Google ScholarDigital Library
6. Xuejin Chen, Sing Bing Kang, Ying-Qing Xu, Julie Dorsey, and Heung-Yeung Shum. 2008. Sketching reality: Realistic interpretation of architectural designs. Trans. on Graphics 27 (2008). Issue 2.Google Scholar
7. Frederic Cordier, Hyewon Seo, Mahmoud Melkemi, and Nickolas S. Sapidis. 2013. Inferring Mirror Symmetric 3D Shapes from Sketches. Computer Aided Design 45, 2 (Feb. 2013), 301–311.Google ScholarDigital Library
8. Johanna Delanoy, Mathieu Aubry, Phillip Isola, Alexei Efros, and Adrien Bousseau. 2018. 3D Sketching using Multi-View Deep Volumetric Prediction. Proceedings of the ACM on Computer Graphics and Interactive Techniques 1, 21 (2018).Google ScholarDigital Library
9. Julie Dorsey, Songhua Xu, Gabe Smedresman, Holly Rushmeier, and Leonard McMillan. 2007. The Mental Canvas: A Tool for Conceptual Architectural Design and Analysis. In Proc. IEEE Pacific Conference on Computer Graphics and Applications. 201–210.Google ScholarDigital Library
10. Koos Eissen and Roselien Steur. 2008. Sketching: Drawing Techniques for Product Designers. Bis Publishers.Google Scholar
11. Koos Eissen and Roselien Steur. 2011. Sketching: The Basics. Bis Publishers.Google Scholar
12. Jean-Dominique Favreau, Florent Lafarge, and Adrien Bousseau. 2016. Fidelity vs. simplicity: a global approach to line drawing vectorization. ACM Transactions on Graphics (TOG) 35, 4 (2016), 120.Google ScholarDigital Library
13. Yotam Gingold, Takeo Igarashi, and Denis Zorin. 2009. Structured annotations for 2D-to-3D modeling. In ACM Trans. Graph. (Proc. SIGGRAPH Asia), Vol. 28.Google Scholar
14. Yulia Gryaditskaya, Mark Sypesteyn, Jan Willem Hoftijzer, Sylvia Pont, Fredo Durand, and Adrien Bousseau. 2019. OpenSketch: A Richly-Annotated Dataset of Product Design Sketches. ACM Transactions on Graphics (Proc. SIGGRAPH Asia) (2019).Google Scholar
15. Erwan Guillou, Daniel Meneveaux, Eric Maisel, and Kadi Bouatouch. 2000. Using vanishing points for camera calibration and coarse 3D reconstruction from a single image. The Visual Computer 16, 7 (2000).Google Scholar
16. Varsha Hedau, Derek Hoiem, and David Forsyth. 2009. Recovering the spatial layout of cluttered rooms. In Proc. ICCV. IEEE.Google ScholarCross Ref
17. Derek Hoiem, Alexei A. Efros, and Martial Hebert. 2005. Automatic Photo Pop-Up. In ACM Transactions on Graphics (Proc. SIGGRAPH). 577–584.Google Scholar
18. Kun Huang, Yifan Wang, Zihan Zhou, Tianjiao Ding, Shenghua Gao, and Yi Ma. 2018. Learning to Parse Wireframes in Images of Man-Made Environments. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google ScholarCross Ref
19. Takeo Igarashi, Satoshi Matsuoka, and Hidehiko Tanaka. 1999. Teddy: A Sketching Interface for 3D Freeform Design. SIGGRAPH (1999).Google ScholarDigital Library
20. Dong Kang, Mark Masry, and Hod Lipson. 2004. Reconstruction of a 3D object from a main axis system. AAAI Fall Symposium – Making Pen-Based Interaction Intelligent and Natural (2004).Google Scholar
21. Changjian Li, Hao Pan, Yang Liu, Alla Sheffer, and Wenping Wang. 2017. BendSketch: Modeling Freeform Surfaces Through 2D Sketching. ACM Trans. Graph. (Proc. SIGGRAPH) 36, 4 (2017).Google ScholarDigital Library
22. Changjian Li, Hao Pan, Yang Liu, Xin Tong, Alla Sheffer, and Wenping Wang. 2018. Robust flow-guided neural prediction for sketch-based freeform surface modeling. In ACM Trans. Graph. (Proc. SIGGRAPH Asia). ACM, 238.Google ScholarDigital Library
23. H Lipson and M Shpitalni. 1996. Optimization-based reconstruction of a 3D object from a single freehand line drawing. Computer-Aided Design 28, 8 (1996), 651 — 663. Google ScholarCross Ref
24. Chenxi Liu, Enrique Rosales, and Alla Sheffer. 2018. StrokeAggregator: consolidating raw sketches into artist-intended curve drawings. ACM Transactions on Graphics (TOG) 37, 4 (2018), 97.Google ScholarDigital Library
25. Jianzhuang Liu, Liangliang Cao, Zhenguo Li, and Xiaoou Tang. 2008. Plane-based optimization for 3D object reconstruction from single line drawings. IEEE Trans. Pattern Anal. Mach. Intell. 30, 2 (2008), 315–327.Google ScholarDigital Library
26. Xueting Liu, Xiangyu Mao, Xuan Yang, Linling Zhang, and Tien-Tsin Wong. 2013. Stereoscopizing Cel Animations. ACM Transactions on Graphics (Proc. SIGGRAPH Asia) 32, 6 (November 2013), 223:1–223:10.Google ScholarDigital Library
27. Xueting Liu, Tien-Tsin Wong, and Pheng-Ann Heng. 2015. Closure-aware Sketch Simplification. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 34, 6 (November 2015).Google ScholarDigital Library
28. Zhaoliang Lun, Matheus Gadelha, Evangelos Kalogerakis, Subhransu Maji, and Rui Wang. 2017. 3D shape reconstruction from sketches via multi-view convolutional networks. In IEEE International Conference on 3D Vision (3DV). 67–77.Google ScholarCross Ref
29. Andrew Nealen, Takeo Igarashi, Olga Sorkine, and Marc Alexa. 2007. FiberMesh: designing freeform surfaces with 3D curves. ACM transactions on graphics (TOG) 26, 3 (2007), 41.Google ScholarDigital Library
30. Simon Niklaus, Long Mai, Jimei Yang, and Feng Liu. 2019. 3D Ken Burns Effect from a Single Image. ACM Transactions on Graphics (Proc. SIGGRAPH Asia) 38, 6, Article 184 (Nov. 2019), 15 pages.Google Scholar
31. Pauline Olivier, Renaud Chabrier, Damien Rohmer, Eric De Thoisy, and Marie-Paule Cani. 2019. Nested Explorative Maps: A new 3D canvas for conceptual design in architecture. Computers and Graphics (Proc. SMI) 82 (2019).Google Scholar
32. GüNay Orbay and Levent Burak Kara. 2012. Sketch-based surface design using malleable curve networks. Comput. Graph. Forum 36, 8 (2012).Google Scholar
33. Radu Orghidan, Joaquim Salvi, Mihaela Gordan, and Bogdan Orza. 2012. Camera calibration using two or three vanishing points. In FedCSIS.Google Scholar
34. Patrick Paczkowski, Min H. Kim, Yann Morvan, Julie Dorsey, Holly Rushmeier, and Carol O’Sullivan. 2011. Insitu: Sketching Architectural Designs in Context. ACM Transactions on Graphics (Proc. SIGGRAPH Asia) 30, 6 (2011).Google Scholar
35. Scott Robertson and Thomas Bertling. 2013. How to Draw: drawing and sketching objects and environments from your imagination. Design Studio Press.Google Scholar
36. Ryan Schmidt, Azam Khan, Gord Kurtenbach, and Karan Singh. 2009a. On Expert Performance in 3D Curve-Drawing Tasks. In Proc. Sketch-Based Interfaces and Modeling.Google ScholarDigital Library
37. Ryan Schmidt, Azam Khan, Karan Singh, and Gord Kurtenbach. 2009b. Analytic drawing of 3D scaffolds. In ACM transactions on graphics (TOG), Vol. 28. ACM, 149.Google Scholar
38. Cloud Shao, Adrien Bousseau, Alla Sheffer, and Karan Singh. 2011. CrossShade: shading concept sketches using cross-section curves. ACM Transactions on Graphics 31, 4 (2011).Google Scholar
39. Tianjia Shao, Wilmot Li, Kun Zhou, Weiwei Xu, Baining Guo, and Niloy J. Mitra. 2013. Interpreting Concept Sketches. ACM Transactions on Graphics (Proc. SIGGRAPH) 32, 4 (2013), 10.Google ScholarDigital Library
40. Alex Shtof, Alexander Agathos, Yotam Gingold, Ariel Shamir, and Daniel Cohen-Or. 2013. Geosemantic Snapping for Sketch-Based Modeling. Computer Graphics Forum 32, 2 (2013), 245–253.Google ScholarCross Ref
41. Edgar Simo-Serra, Satoshi Iizuka, Kazuma Sasaki, and Hiroshi Ishikawa. 2016. Learning to Simplify: Fully Convolutional Networks for Rough Sketch Cleanup. ACM Trans. Graph. (Proc. SIGGRAPH) 35, 4 (2016).Google ScholarDigital Library
42. Daniel Sýkora, Ladislav Kavan, Martin Čadík, Ondřej Jamriška, Alec Jacobson, Brian Whited, Maryann Simmons, and Olga Sorkine-Hornung. 2014. Ink-and-Ray: BasRelief Meshes for Adding Global Illumination Effects to Hand-Drawn Characters. ACM Transaction on Graphics 33, 2 (2014), 16.Google ScholarDigital Library
43. Chao Tian, Mark A. Masry, and Hod Lipson. 2009. Physical sketching: Reconstruction and analysis of 3D objects from freehand sketches. Computer Aided Dessign 41 (2009), 147–158.Google ScholarDigital Library
44. Anil Usumezbas, Ricardo Fabbri, and Benjamin B. Kimia. 2016. From multiview image curves to 3D drawings. In Proc. European Conference on Computer Vision (ECCV).Google Scholar
45. Yingze Wang, Yu Chen, Jianzhuang Liu, and Xiaoou Tang. 2009. 3D reconstruction of curved objects from single 2D line drawings. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
46. Baoxuan Xu, William Chang, Alla Sheffer, Adrien Bousseau, James McCrae, and Karan Singh. 2014. True2Form: 3D curve networks from 2D sketches via selective regularization. ACM Transactions on Graphics 33, 4 (2014).Google ScholarDigital Library
47. Robert C. Zeleznik, Kenneth P. Herndon, and John F. Hughes. 1996. SKETCH: An Interface for Sketching 3D Scenes. SIGGRAPH (1996).Google Scholar
48. Yichao Zhou, Haozhi Qi, Yuexiang Zhai, Qi Sun, Zhili Chen, Li-Yi Wei, and Yi Ma. 2019. Learning to Reconstruct 3D Manhattan Wireframes From a Single Image. In Proc. IEEE International Conference on Computer Vision (ICCV).Google ScholarCross Ref
