“A benchmark for rough sketch cleanup” by Yan, Vanderhaeghe and Gingold
Conference:
Type(s):
Title:
- A benchmark for rough sketch cleanup
Session/Category Title: All About Sketches
Presenter(s)/Author(s):
Abstract:
Sketching is a foundational step in the design process. Decades of sketch processing research have produced algorithms for 3D shape interpretation, beautification, animation generation, colorization, etc. However, there is a mismatch between sketches created in the wild and the clean, sketch-like input required by these algorithms, preventing their adoption in practice. The recent flurry of sketch vectorization, simplification, and cleanup algorithms could be used to bridge this gap. However, they differ wildly in the assumptions they make on the input and output sketches. We present the first benchmark to evaluate and focus sketch cleanup research. Our dataset consists of 281 sketches obtained in the wild and a curated subset of 101 sketches. For this curated subset along with 40 sketches from previous work, we commissioned manual vectorizations and multiple ground truth cleaned versions by professional artists. The sketches span artistic and technical categories and were created by a variety of artists with different styles. Most sketches have Creative Commons licenses; the rest permit academic use. Our benchmark’s metrics measure the similarity of automatically cleaned rough sketches to artist-created ground truth; the ambiguity and messiness of rough sketches; and low-level properties of the output parameterized curves. Our evaluation identifies shortcomings among state-of-the-art cleanup algorithms and discusses open problems for future research.
References:
1. Alexis Andre and Suguru Saito. 2011. Single-view sketch based modeling. In Proceedings of Sketch-Based Interfaces and Modeling (SBIM). Association for Computing Machinery, Vancouver, British Columbia, Canada, 133–140. Google ScholarDigital Library
2. Seok-Hyung Bae, Ravin Balakrishnan, and Karan Singh. 2008. ILoveSketch: As-natural-as-possible sketching system for creating 3D curve models. In Proceedings of ACM UIST. 151–160. Google ScholarDigital Library
3. Pascal Barla, Joëlle Thollot, and François X. Sillion. 2005. Geometric Clustering for Line Drawing Simplification. In Proceedings of the Eurographics Symposium on Rendering (EGSR), Oliver Deussen, Alexander Keller, Kavita Bala, Philip Dutré, Dieter W. Fellner, and Stephen N. Spencer (Eds.). Konstanz, Germany, 183–192. https://hal.inria.fr/inria-00362893Google Scholar
4. Mikhail Bessmeltsev, Will Chang, Nicholas Vining, Alla Sheffer, and Karan Singh. 2015. Modeling Character Canvases from Cartoon Drawings. ACM Trans. Graph. 34, 5, Article 162 (Nov. 2015), 16 pages. Google ScholarDigital Library
5. Mikhail Bessmeltsev and Justin Solomon. 2019. Vectorization of Line Drawings via Polyvector Fields. ACM Transactions on Graphics 38, 1 (Jan. 2019), 1–12. Google ScholarDigital Library
6. Gwern Branwen. 2019. Danbooru2019: A Large-Scale Crowdsourced and Tagged Anime Illustration Dataset. https://www.gwern.net/Danbooru2019. Accessed: 2020-01-15.Google Scholar
7. Boris Dalstein, Rémi Ronfard, and Michiel van de Panne. 2014. Vector Graphics Complexes. ACM Transactions on Graphics 33, 4 (July 2014).Google ScholarDigital Library
8. Luca Donati, Simone Cesano, and Andrea Prati. 2019. A complete hand-drawn sketch vectorization framework. Multimedia Tools and Applications 78, 14 (July 2019), 19083–19113. Google ScholarDigital Library
9. Mathias Eitz, James Hays, and Marc Alexa. 2012. How do humans sketch objects? ACM Trans. Graph. 31, 4, Article 44 (July 2012), 10 pages. Google ScholarDigital Library
10. Jean-Dominique Favreau, Florent Lafarge, and Adrien Bousseau. 2016. Fidelity vs. simplicity: a global approach to line drawing vectorization. ACM Transactions on Graphics 35, 4 (July 2016), 1–10. Google ScholarDigital Library
11. Jakub Fišer, Paul Asente, Stephen Schiller, and Daniel Sýkora. 2016. Advanced Drawing Beautification with ShipShape. Computers & Graphics 56 (2016), 46–58.Google ScholarDigital Library
12. Sarah Frisken. 2008. Efficient curve fitting. Journal of Graphics Tools 13, 2 (2008), 37–54.Google ScholarCross Ref
13. Cindy Grimm and Pushkar Joshi. 2012. Just DrawIt: A 3D Sketching System. In Proceedings of the International Symposium on Sketch-Based Interfaces and Modeling (Annecy, France) (SBIM ’12). Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, 121–130. http://dl.acm.org/citation.cfm?id=2331067.2331084Google Scholar
14. Yulia Gryaditskaya, Mark Sypesteyn, Jan Willem Hoftijzer, Sylvia Pont, Frédo Durand, and Adrien Bousseau. 2019. OpenSketch: A Richly-Annotated Dataset of Product Design Sketches. ACM Transactions on Graphics (SIGGRAPH Asia Conference Proceedings) 38, 6 (Nov. 2019). http://www-sop.inria.fr/reves/Basilic/2019/GSHPDB19Google Scholar
15. David Ha and Douglas Eck. 2018. A Neural Representation of Sketch Drawings. In ICLR 2018. https://openreview.net/pdf?id=Hy6GHpkCWGoogle Scholar
16. Takeo Igarashi, Sachiko Kawachiya, Hidehiko Tanaka, and Satoshi Matsuoka. 1998. Pegasus: A drawing system for rapid geometric design. In Proceedings of ACM SIGCHI (Los Angeles, California, United States). 24–25. Google ScholarDigital Library
17. Matthew Kaplan and Elaine Cohen. 2006. Producing models from drawings of curved surfaces. In Proceedings of Sketch-Based Interfaces and Modeling (SBIM). Eurographics Association, Vienna, Austria, 51–59.Google Scholar
18. Byungsoo Kim, Oliver Wang, A. Cengiz Öztireli, and Markus Gross. 2018. Semantic Segmentation for Line Drawing Vectorization Using Neural Networks. Computer Graphics Forum 37, 2 (2018), 329–338. Google ScholarCross Ref
19. Andrew Kriebel. 2017. How Creating a Design Funnel Can Lead to Better Design Decisions. https://mindtribe.com/2017/11/varying-the-design-space/Google Scholar
20. James A. Landay and Brad A. Myers. 1995. Interactive Sketching for the Early Stages of User Interface Design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’95). ACM Press/Addison-Wesley Publishing Co., USA, 43–50. Google ScholarDigital Library
21. Hod Lipson and Moshe Shpitalni. 1996. Optimization-based reconstruction of a 3D object from a single freehand line drawing. Computer-Aided Design 28, 8 (1996), 651–663. Google ScholarCross Ref
22. Chenxi Liu, Enrique Rosales, and Alla Sheffer. 2018. StrokeAggregator: consolidating raw sketches into artist-intended curve drawings. ACM Transactions on Graphics 37, 4 (July 2018), 1–15. Google ScholarDigital Library
23. Xueting Liu, Tien-Tsin Wong, and Pheng-Ann Heng. 2015. Closure-aware sketch simplification. ACM Transactions on Graphics 34, 6 (Oct. 2015), 1–10. Google ScholarDigital Library
24. Yue Liu, Xuemei Li, Pengbo Bo, and Xifeng Gao. 2019. Sketch simplification guided by complex agglomeration. Science China Information Sciences 62, 5 (April 2019), 52105. Google ScholarCross Ref
25. Yusuke Matsui, Kota Ito, Yuji Aramaki, Azuma Fujimoto, Toru Ogawa, Toshihiko Yamasaki, and Kiyoharu Aizawa. 2017. Sketch-based manga retrieval using Manga109 dataset. Multimedia Tools and Applications 76, 20 (2017), 21811–21838.Google ScholarDigital Library
26. Damien Newman. 2002. The Process of Design Squiggle. https://thedesignsquiggle.com/Google Scholar
27. Gioacchino Noris, Alexander Hornung, Robert W. Sumner, Maryann Simmons, and Markus Gross. 2013. Topology-driven vectorization of clean line drawings. ACM Transactions on Graphics 32, 1 (Jan. 2013), 1–11. Google ScholarDigital Library
28. Toru Ogawa, Yusuke Matsui, Toshihiko Yamasaki, and Kiyoharu Aizawa. 2016. Sketch simplification by classifying strokes. In 2016 23rd International Conference on Pattern Recognition (ICPR). 1065–1070. ISSN: null. Google ScholarCross Ref
29. Günay Orbay and Levent Burak Kara. 2011. Beautification of Design Sketches Using Trainable Stroke Clustering and Curve Fitting. IEEE Transactions on Visualization and Computer Graphics 17, 5 (May 2011), 694–708. Google ScholarDigital Library
30. Amal Dev Parakkat, Uday Bondi Pundarikaksha, and Ramanathan Muthuganapathy. 2018. A Delaunay triangulation based approach for cleaning rough sketches. Computers & Graphics 74 (Aug. 2018), 171–181. Google ScholarCross Ref
31. Theo Pavlidis and Christopher J. Van Wyk. 1985. An Automatic Beautifier for Drawings and Illustrations. In Proceedings of the 12th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’85). Association for Computing Machinery, New York, NY, USA, 225–234. Google ScholarDigital Library
32. Patsorn Sangkloy, Nathan Burnell, Cusuh Ham, and James Hays. 2016. The Sketchy Database: Learning to Retrieve Badly Drawn Bunnies. ACM Transactions on Graphics (proceedings of SIGGRAPH) (2016).Google ScholarDigital Library
33. Cloud Shao, Adrien Bousseau, Alla Sheffer, and Karan Singh. 2012. CrossShade: shading concept sketches using cross-section curves. ACM Trans. Graph. 31, 4, Article 45 (July 2012), 11 pages. Google ScholarDigital Library
34. Tianjia Shao, Wilmot Li, Kun Zhou, Weiwei Xu, Baining Guo, and Niloy J. Mitra. 2013. Interpreting Concept Sketches. ACM Transactions on Graphics 32, 4 (2013), 10.Google ScholarDigital Library
35. Amit Shesh and Baoquan Chen. 2008. Efficient and Dynamic Simplification of Line Drawings. Computer Graphics Forum 27, 2 (2008), 537–545. Google ScholarCross Ref
36. Alex Shtof, Alexander Agathos, Yotam Gingold, Ariel Shamir, and Daniel Cohen-Or. 2013. Geosemantic Snapping for Sketch-Based Modeling. Computer Graphics Forum 32, 2 (2013), 245–253. Google ScholarCross Ref
37. Edgar Simo-Serra, Satoshi Iizuka, and Hiroshi Ishikawa. 2018a. Mastering Sketching: Adversarial Augmentation for Structured Prediction. ACM Transactions on Graphics 37, 1 (Jan. 2018), 1–13. Google ScholarDigital Library
38. Edgar Simo-Serra, Satoshi Iizuka, and Hiroshi Ishikawa. 2018b. Real-Time Data-Driven Interactive Rough Sketch Inking. ACM Transactions on Graphics (SIGGRAPH) 37, 4 (2018).Google ScholarDigital Library
39. Edgar Simo-Serra, Satoshi Iizuka, Kazuma Sasaki, and Hiroshi Ishikawa. 2016. Learning to Simplify: Fully Convolutional Networks for Rough Sketch Cleanup. ACM Transactions on Graphics (SIGGRAPH) 35, 4 (2016).Google ScholarDigital Library
40. Ivan Edward Sutherland. 1963. Sketchpad: A man-machine graphical communication system. Ph.D. Dissertation. Massachusetts Institute of Technology.Google Scholar
41. Brian Whited, Gioacchino Noris, Maryann Simmons, Robert W. Sumner, Markus Gross, and Jarek Rossignac. 2010. BetweenIT: An Interactive Tool for Tight Inbetweening. Computer Graphics Forum 29, 2 (2010), 605–614. Google ScholarCross Ref
42. Baoxuan Xu, William Chang, Alla Sheffer, Adrien Bousseau, James McCrae, and Karan Singh. 2014. True2Form: 3D Curve Networks from 2D Sketches via Selective Regularization. ACM Trans. Graph. 33, 4, Article 131 (July 2014), 13 pages. Google ScholarDigital Library
43. Xuemiao Xu, Minshan Xie, Peiqi Miao, Wei Qu, Wenpeng Xiao, Huaidong Zhang, Xueting Liu, and Tien-Tsin Wong. 2019. Perceptual-aware Sketch Simplification Based on Integrated VGG Layers. IEEE Transactions on Visualization and Computer Graphics (2019). Conference Name: IEEE Transactions on Visualization and Computer Graphics. Google ScholarDigital Library
44. Wenwu Yang, Hock-Soon Seah, Quan Chen, Hong-Ze Liew, and Daniel Sýkora. 2018. FTP-SC: Fuzzy Topology Preserving Stroke Correspondence. Computer Graphics Forum 37, 8 (2018), 125–135. Google ScholarCross Ref
45. Youyi Zheng, Han Liu, Julie Dorsey, and Niloy J. Mitra. 2016. SmartCanvas: Context-inferred Interpretation of Sketches for Preparatory Design Studies. Computer Graphics Forum 35, 2 (May 2016), 37–48.Google ScholarCross Ref
