“Real-time data-driven interactive rough sketch inking” by Simo-Serra, Iizuka and Ishikawa
Conference:
Type:
Entry Number: 98
Title:
- Real-time data-driven interactive rough sketch inking
Session/Category Title: Sketching
Presenter(s)/Author(s):
Moderator(s):
Abstract:
We present an interactive approach for inking, which is the process of turning a pencil rough sketch into a clean line drawing. The approach, which we call the Smart Inker, consists of several “smart” tools that intuitively react to user input, while guided by the input rough sketch, to efficiently and naturally connect lines, erase shading, and fine-tune the line drawing output. Our approach is data-driven: the tools are based on fully convolutional networks, which we train to exploit both the user edits and inaccurate rough sketch to produce accurate line drawings, allowing high-performance interactive editing in real-time on a variety of challenging rough sketch images. For the training of the tools, we developed two key techniques: one is the creation of training data by simulation of vague and quick user edits; the other is a line normalization based on learning from vector data. These techniques, in combination with our sketch-specific data augmentation, allow us to train the tools on heterogeneous data without actual user interaction. We validate our approach with an in-depth user study, comparing it with professional illustration software, and show that our approach is able to reduce inking time by a factor of 1.8X, while improving the results of amateur users.
References:
1. Maher Ahmed and Rabab Ward. 2002. A rotation invariant rule-based thinning algorithm for character recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 24, 12 (2002), 1672–1678. Google ScholarDigital Library
2. Seok-Hyung Bae, Ravin Balakrishnan, and Karan Singh. 2008. ILoveSketch: As-natural-as-possible Sketching System for Creating 3D Curve Models. In ACM Symposium on User Interface Software and Technology. 151–160. Google ScholarDigital Library
3. Houssem Chatbri and Keisuke Kameyama. 2014. Using scale space filtering to make thinning algorithms robust against noise in sketch images. Pattern Recognition Letters 42 (2014), 1–10.Google ScholarCross Ref
4. Jiazhou Chen, Gaël Guennebaud, Pascal Barla, and Xavier Granier. 2013. Non-Oriented MLS Gradient Fields. Computer Graphics Forum 32, 8 (2013), 98–109.Google ScholarCross Ref
5. David H Douglas and Thomas K Peucker. 1973. Algorithms for the reduction of the number of points required to represent a digitized line or its caricature. Cartographica: The International Journal for Geographic Information and Geovisualization 10, 2 (1973), 112–122.Google ScholarCross Ref
6. Charles R Dyer and Azriel Rosenfeld. 1979. Thinning algorithms for gray-scale pictures. IEEE Transactions on Pattern Analysis and Machine Intelligence 1 (1979), 88–89. Google ScholarDigital Library
7. Mathias Eitz, James Hays, and Marc Alexa. 2012. How Do Humans Sketch Objects? ACM Transactions on Graphics (Proceedings of SIGGRAPH) 31, 4 (2012), 44:1–44:10. Google ScholarDigital Library
8. Jean-Dominique Favreau, Florent Lafarge, and Adrien Bousseau. 2016. Fidelity vs. Simplicity: a Global Approach to Line Drawing Vectorization. ACM Transactions on Graphics (Proceedings of SIGGRAPH) 35, 4 (2016). Google ScholarDigital Library
9. Jakub Fišer, Paul Asente, Stephen Schiller, and Daniel Sýkora. 2015. ShipShape: A Drawing Beautification Assistant. In Workshop on Sketch-Based Interfaces and Modeling. 49–57. Google ScholarDigital Library
10. Cindy Grimm and Pushkar Joshi. 2012. Just Drawlt: A 3D Sketching System. In nternational Symposium on Sketch-Based Interfaces and Modeling. 121–130. Google ScholarDigital Library
11. David Ha and Douglas Eck. 2018. A Neural Representation of Sketch Drawings. In International Conference on Learning Representations.Google Scholar
12. Xavier Hilaire and Karl Tombre. 2006. Robust and accurate vectorization of line drawings. IEEE Transactions on Pattern Analysis and Machine Intelligence 28, 6 (2006), 890–904. Google ScholarDigital Library
13. Takeo Igarashi, Satoshi Matsuoka, Sachiko Kawachiya, and Hidehiko Tanaka. 1997. Interactive Beautification: A Technique for Rapid Geometric Design. In ACM Symposium on User Interface Software and Technology. 105–114. Google ScholarDigital Library
14. Sergey Ioffe and Christian Szegedy. 2015. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In International Conference on Machine Learning. Google ScholarDigital Library
15. Jang and Roland T. Chin. 1990. Analysis of thinning algorithms using mathematical morphology. IEEE Transactions on Pattern Analysis and Machine Intelligence 12, 6 (1990), 541–551. Google ScholarDigital Library
16. Louisa Lam, Seong-Whan Lee, and Ching Y Suen. 1992. Thinning methodologies-a comprehensive survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 14, 9 (1992), 869–885. Google ScholarDigital Library
17. David Lindlbauer, Michael Haller, Mark S. Hancock, Stacey D. Scott, and Wolfgang Stuerzlinger. 2013. Perceptual grouping: selection assistance for digital sketching. In International Conference on Interactive Tabletops and Surfaces. 51–60. Google ScholarDigital Library
18. Xueting Liu, Tien-Tsin Wong, and Pheng-Ann Heng. 2015. Closure-aware Sketch Simplification. ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia) 34, 6 (2015), 168:1–168:10. Google ScholarDigital Library
19. Henry B Mann and Donald R Whitney. 1947. On a test of whether one of two random variables is stochastically larger than the other. The annals of mathematical statistics (1947), 50–60.Google Scholar
20. Gary Martin, Steve Rude, and Terry Austin. 1997. The Art of Comic Book Inking. Dark Horse Comics.Google Scholar
21. Ana Maria Mendonca and Aurelio Campilho. 2006. Segmentation of retinal blood vessels by combining the detection of centerlines and morphological reconstruction. IEEE Transactions on Medical Imaging 25, 9 (2006), 1200–1213.Google ScholarCross Ref
22. Vinod Nair and Geoffrey E Hinton. 2010. Rectified linear units improve restricted boltzmann machines. In International Conference on Machine Learning. 807–814. Google ScholarDigital Library
23. Gioacchino Noris, Alexander Hornung, Robert W Sumner, Maryann Simmons, and Markus Gross. 2013. Topology-driven Vectorization of Clean Line Drawings. ACM Transactions on Graphics 32, 1 (2013), 4:1–4:11. Google ScholarDigital Library
24. Günay Orbay and Levent Burak Kara. 2011. Beautification of Design Sketches Using Trainable Stroke Clustering and Curve Fitting. IEEE Transactions on Visualization and Computer Graphics 17, 5 (2011), 694–708. Google ScholarDigital Library
25. Urs Ramer. 1972. An iterative procedure for the polygonal approximation of plane curves. Computer graphics and image processing 1, 3 (1972), 244–256.Google Scholar
26. Patsorn Sangkloy, Jingwan Lu, Chen Fang, Fisher Yu, and James Hays. 2017. Scribbler: Controlling Deep Image Synthesis with Sketch and Color. In IEEE Conference on Computer Vision and Pattern Recognition.Google Scholar
27. Amit Shesh and Baoquan Chen. 2008. Efficient and Dynamic Simplification of Line Drawings. Computer Graphics Forum 27, 2 (2008), 537–545.Google ScholarCross Ref
28. Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. 2016. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In IEEE Conference on Computer Vision and Pattern Recognition.Google ScholarCross Ref
29. Edgar Simo-Serra, Satoshi Iizuka, and Hiroshi Ishikawa. 2018. Mastering Sketching: Adversarial Augmentation for Structured Prediction. ACM Transactions on Graphics 37, 1 (2018). Google ScholarDigital Library
30. Edgar Simo-Serra, Satoshi Iizuka, Kazuma Sasaki, and Hiroshi Ishikawa. 2016. Learning to Simplify: Fully Convolutional Networks for Rough Sketch Cleanup. ACM Transactions on Graphics (Proceedings of SIGGRAPH) 35, 4 (2016). Google ScholarDigital Library
31. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Research 15 (2014), 1929–1958. Google ScholarDigital Library
32. Jonathan Tompson, Ross Goroshin, Arjun Jain, Yann LeCun, and Christoph Bregler. 2015. Efficient object localization using convolutional networks. In IEEE Conference on Computer Vision and Pattern Recognition.Google ScholarCross Ref
33. Jun Xie, Holger Winnemöller, Wilmot Li, and Stephen Schiller. 2017. Interactive Vectorization. In ACM CHI Conference on Human Factors in Computing Systems. Google ScholarDigital Library
34. Matthew D. Zeiler. 2012. ADADELTA: An Adaptive Learning Rate Method. arXiv preprint arXiv:1212.5701 (2012).Google Scholar
35. Richard Zhang, Jun-Yan Zhu, Phillip Isola, Xinyang Geng, Angela S Lin, Tianhe Yu, and Alexei A Efros. 2017. Real-Time User-Guided Image Colorization with Learned Deep Priors. ACM Transactions on Graphics (Proceedings of SIGGRAPH) 9, 4 (2017). Google ScholarDigital Library
36. TY Zhang and Ching Y. Suen. 1984. A fast parallel algorithm for thinning digital patterns. Commun. ACM 27, 3 (1984), 236–239. Google ScholarDigital Library
