“Learning to simplify: fully convolutional networks for rough sketch cleanup”
Conference:
Type(s):
Title:
- Learning to simplify: fully convolutional networks for rough sketch cleanup
Abstract:
In this paper, we present a novel technique to simplify sketch drawings based on learning a series of convolution operators. In contrast to existing approaches that require vector images as input, we allow the more general and challenging input of rough raster sketches such as those obtained from scanning pencil sketches. We convert the rough sketch into a simplified version which is then amendable for vectorization. This is all done in a fully automatic way without user intervention. Our model consists of a fully convolutional neural network which, unlike most existing convolutional neural networks, is able to process images of any dimensions and aspect ratio as input, and outputs a simplified sketch which has the same dimensions as the input image. In order to teach our model to simplify, we present a new dataset of pairs of rough and simplified sketch drawings. By leveraging convolution operators in combination with efficient use of our proposed dataset, we are able to train our sketch simplification model. Our approach naturally overcomes the limitations of existing methods, e.g., vector images as input and long computation time; and we show that meaningful simplifications can be obtained for many different test cases. Finally, we validate our results with a user study in which we greatly outperform similar approaches and establish the state of the art in sketch simplification of raster images.
References:
1. Bae, S.-H., Balakrishnan, R., and Singh, K. 2008. Ilovesketch: As-natural-as-possible sketching system for creating 3d curve models. In ACM Symposium on User Interface Software and Technology, 151–160. Google ScholarDigital Library
2. Barla, P., Thollot, J., and Sillion, F. X. 2005. Geometric clustering for line drawing simplification. In ACM SIGGRAPH 2005 Sketches. Google ScholarDigital Library
3. Bartolo, A., Camilleri, K. P., Fabri, S. G., Borg, J. C., and Farrugia, P. J. 2007. Scribbles to vectors: Preparation of scribble drawings for cad interpretation. In Eurographics Workshop on Sketch-based Interfaces and Modeling, 123–130. Google ScholarDigital Library
4. Baudel, T. 1994. A mark-based interaction paradigm for freehand drawing. In ACM Symposium on User Interface Software and Technology, 185–192. Google ScholarDigital Library
5. Chang, H.-H., and Yan, H. 1998. Vectorization of hand-drawn image using piecewise cubic bézier curves fitting. Pattern Recognition 31, 11, 1747–1755.Google ScholarCross Ref
6. Chen, J., Guennebaud, G., Barla, P., and Granier, X. 2013. Non-oriented mls gradient fields. Computer Graphics Forum 32, 8, 98–109.Google ScholarCross Ref
7. Cole, F., DeCarlo, D., Finkelstein, A., Kin, K., Morley, K., and Santella, A. 2006. Directing gaze in 3d models with stylized focus. In Eurographics Conference on Rendering Techniques, 377–387. Google ScholarDigital Library
8. Deussen, O., and Strothotte, T. 2000. Computer-generated pen-and-ink illustration of trees. In Conference on Computer Graphics and Interactive Techniques, 13–18. Google ScholarDigital Library
9. Dong, C., Loy, C. C., He, K., and Tang, X. 2016. Image super-resolution using deep convolutional networks. PAMI 38, 2, 295–307. Google ScholarDigital Library
10. Dosovitskiy, A., Springenberg, J. T., and Brox, T. 2015. Learning to generate chairs with convolutional neural networks. In CVPR.Google Scholar
11. Fischer, P., Dosovitskiy, A., Ilg, E., Häusser, P., Hazirbas, C., Golkov, V., van der Smagt, P., Cremers, D., and Brox, T. 2015. Flownet: Learning optical flow with convolutional networks.Google Scholar
12. Fišer, J., Asente, P., and Sýkora, D. 2015. Shipshape: A drawing beautification assistant. In Workshop on Sketch-Based Interfaces and Modeling, 49–57. Google ScholarDigital Library
13. Freeman, H. 1974. Computer processing of line-drawing images. ACM Comput. Surv. 6, 1, 57–97. Google ScholarDigital Library
14. Fukushima, K. 1988. Neocognitron: A hierarchical neural network capable of visual pattern recognition. Neural networks 1, 2, 119–130.Google Scholar
15. Grabli, S., Durand, F., and Sillion, F. 2004. Density measure for line-drawing simplification. In Pacific Conference on Computer Graphics and Applications, 2004, 309–318. Google ScholarDigital Library
16. Grimm, C., and Joshi, P. 2012. Just drawit: A 3d sketching system. In nternational Symposium on Sketch-Based Interfaces and Modeling, 121–130. Google ScholarDigital Library
17. Hilaire, X., and Tombre, K. 2006. Robust and accurate vectorization of line drawings. PAMI 28, 6, 890–904. Google ScholarDigital Library
18. Igarashi, T., Matsuoka, S., Kawachiya, S., and Tanaka, H. 1997. Interactive beautification: A technique for rapid geometric design. In ACM Symposium on User Interface Software and Technology, 105–114. Google ScholarDigital Library
19. Ioffe, S., and Szegedy, C. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML.Google Scholar
20. Janssen, R. D., and Vossepoel, A. M. 1997. Adaptive vectorization of line drawing images. Computer Vision and Image Understanding 65, 1, 38–56. Google ScholarDigital Library
21. Krizhevsky, A., Sutskever, I., and Hinton, G. E. 2012. Imagenet classification with deep convolutional neural networks. In NIPS.Google Scholar
22. LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. 1998. Gradient-based learning applied to document recognition. Proceedings of the IEEE 86, 11, 2278–2324.Google ScholarCross Ref
23. Lindlbauer, D., Haller, M., Hancock, M. S., Scott, S. D., and Stuerzlinger, W. 2013. Perceptual grouping: selection assistance for digital sketching. In International Conference on Interactive Tabletops and Surfaces, 51–60. Google ScholarDigital Library
24. Liu, X., Wong, T.-T., and Heng, P.-A. 2015. Closure-aware sketch simplification. ACM Trans. Graph. 34, 6, 168:1–168:10. Google ScholarDigital Library
25. Long, J., Shelhamer, E., and Darrell, T. 2015. Fully convolutional networks for semantic segmentation. In CVPR.Google Scholar
26. Nair, V., and Hinton, G. E. 2010. Rectified linear units improve restricted boltzmann machines. In ICML, 807–814.Google Scholar
27. Noh, H., Hong, S., and Han, B. 2015. Learning deconvolution network for semantic segmentation. In ICCV. Google ScholarDigital Library
28. Noris, G., Hornung, A., Sumner, R. W., Simmons, M., and Gross, M. 2013. Topology-driven vectorization of clean line drawings. ACM Trans. Graph. 32, 1, 4:1–4:11. Google ScholarDigital Library
29. Orbay, G., and Kara, L. 2011. Beautification of design sketches using trainable stroke clustering and curve fitting. IEEE Trans. on Visualization and Computer Graphics 17, 5, 694–708. Google ScholarDigital Library
30. Preim, B., and Strothotte, T. 1995. Tuning rendered line-drawings. In Winter School in Computer Graphics, 227–237.Google Scholar
31. Pusch, R., Samavati, F., Nasri, A., and Wyvill, B. 2007. Improving the sketch-based interface. The Visual Computer 23, 9-11, 955–962. Google ScholarDigital Library
32. Rosin, P. L. 1994. Grouping curved lines. In Machine Graphics and Vision 7, 625–644.Google Scholar
33. Rumelhart, D., Hinton, G., and Williams, R. 1986. Learning representations by back-propagating errors. In Nature.Google Scholar
34. Selinger, P. 2003. Potrace: a polygon-based tracing algorithm. Potrace (online), http://potrace. sourceforge. net/potrace.pdf (2009-07-01).Google Scholar
35. Shen, W., Wang, X., Wang, Y., Bai, X., and Zhang, Z. 2015. Deepcontour: A deep convolutional feature learned by positive-sharing loss for contour detection. In CVPR.Google Scholar
36. Shesh, A., and Chen, B. 2008. Efficient and dynamic simplification of line drawings. Computer Graphics Forum 27, 2, 537–545.Google ScholarCross Ref
37. Simonyan, K., and Zisserman, A. 2015. Very deep convolutional networks for large-scale image recognition. In ICLR.Google Scholar
38. Springenberg, J. T., Dosovitskiy, A., Brox, T., and Riedmiller, M. A. 2015. Striving for simplicity: The all convolutional net. In ICLR Workshop Track.Google Scholar
39. Wilson, B., and Ma, K.-L. 2004. Rendering complexity in computer-generated pen-and-ink illustrations. In International Symposium on Non-photorealistic Animation and Rendering, 129–137. Google ScholarDigital Library
40. Zeiler, M. D., and Fergus, R. 2014. Visualizing and understanding convolutional networks. In ECCV.Google Scholar
41. Zeiler, M. D. 2012. ADADELTA: an adaptive learning rate method. CoRR abs/1212.5701.Google Scholar
42. Zhang, T. Y., and Suen, C. Y. 1984. A fast parallel algorithm for thinning digital patterns. Commun. ACM 27, 3, 236–239. Google ScholarDigital Library
