“Deep extraction of manga structural lines”
Conference:
Type:
Title:
- Deep extraction of manga structural lines
Session/Category Title: Deep Image Processing
Presenter(s)/Author(s):
Moderator(s):
Abstract:
Extraction of structural lines from pattern-rich manga is a crucial step for migrating legacy manga to digital domain. Unfortunately, it is very challenging to distinguish structural lines from arbitrary, highly-structured, and black-and-white screen patterns. In this paper, we present a novel data-driven approach to identify structural lines out of pattern-rich manga, with no assumption on the patterns. The method is based on convolutional neural networks. To suit our purpose, we propose a deep network model to handle the large variety of screen patterns and raise output accuracy. We also develop an efficient and effective way to generate a rich set of training data pairs. Our method suppresses arbitrary screen patterns no matter whether these patterns are regular, irregular, tone-varying, or even pictorial, and regardless of their scales. It outputs clear and smooth structural lines even if these lines are contaminated by and immersed in complex patterns. We have evaluated our method on a large number of mangas of various drawing styles. Our method substantially outperforms state-of-the-art methods in terms of visual quality. We also demonstrate its potential in various manga applications, including manga colorization, manga retargeting, and 2.5D manga generation.
References:
1. Pablo Arbelaez, Michael Maire, Charless C. Flowlkes, and Jitendra Malik. 2011. Contour Detection and Hierarchical Image Segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 33, 5 (2011), 898–916. Google ScholarDigital Library
2. Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. 2009. PatchMatch: A Randomized Correspondence Algorithm for Structural Image Editing. ACM Transactions on Graphics (Proc. SIGGRAPH) 28, 3 (Aug. 2009).Google ScholarDigital Library
3. John Canny. 1986. A Computational Approach to Edge Detection. IEEE Trans. Pattern Anal. Mach. Intell. 8, 6 (1986), 679–698. Google ScholarDigital Library
4. François Chollet. 2015. Keras. https://github.com/fchollet/keras. (2015).Google Scholar
5. James Hays, Marius Leordeanu, Alexei A Efros, and Yanxi Liu. 2006. Discovering texture regularity as a higher-order correspondence problem. In Computer Vision-ECCV 2006. Springer, 522–535.Google Scholar
6. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016a. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27–30, 2016. 770–778. Google ScholarCross Ref
7. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016b. Identity Mappings in Deep Residual Networks. In Computer Vision – ECCV 2016 – 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV. 630–645. Google ScholarCross Ref
8. Sergey Ioffe and Christian Szegedy. 2015. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6–11 July 2015. 448–456.Google ScholarDigital Library
9. Phillip Isola, Daniel Zoran, Dilip Krishnan, and Edward H. Adelson. 2014. Crisp Boundary Detection Using Pointwise Mutual Information. In ECCV. Google ScholarCross Ref
10. Kota Ito, Yusuke Matsui, Toshihiko Yamasaki, and Kiyoharu Aizawa. 2015. Separation of Manga Line Drawings and Screentones. In Eurographics 2015 – Short Papers, Zurich, Switzerland, May 4–8, 2015. 73–76.Google Scholar
11. Henry Kang, Seungyong Lee, and Charles K. Chui. 2007. Coherent line drawing. In Proceedings of the 5th International Symposium on Non-Photorealistic Animation and Rendering 2007, San Diego, California, USA, August 4–5, 2007. 43–50. Google ScholarDigital Library
12. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. CoRR abs/1412.6980 (2014).Google Scholar
13. Iasonas Kokkinos. 2015. Pushing the boundaries of boundary detection using deep learning. arXiv preprint arXiv:1511.07386 (2015).Google Scholar
14. Johannes Kopf and Dani Lischinski. 2012. Digital reconstruction of halftoned color comics. ACM Transactions on Graphics (TOG) 31, 6 (2012), 140.Google ScholarDigital Library
15. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Proceedings of a meeting held December 3–6, 2012, Lake Tahoe, Nevada, United States. 1106–1114.Google Scholar
16. Xueting Liu, Xiangyu Mao, Xuan Yang, Linling Zhang, and Tien-Tsin Wong. 2013. Stereoscopizing Cel Animations. ACM Transactions on Graphics (SIGGRAPH Asia 2013 issue) 32, 6 (November 2013), 223:1–223:10.Google Scholar
17. Yanxi Liu, Tamara Belkina, James Hays, and Roberto Lublinerman. 2008. Image Defencing. In Proceedings of CVPR 2008.Google Scholar
18. Yanxi Liu, Robert T Collins, and Yanghai Tsin. 2004a. A computational model for periodic pattern perception based on frieze and wallpaper groups. Pattern Analysis and Machine Intelligence, IEEE Transactions on 26, 3 (2004), 354–371.Google ScholarDigital Library
19. Yanxi Liu, Wen-Chieh Lin, and James Hays. 2004b. Near-regular texture analysis and manipulation. In ACM Transactions on Graphics (TOG), Vol. 23. ACM, 368–376. Google ScholarDigital Library
20. Xiangyu Mao, Xueting Liu, Tien-Tsin Wong, and Xuemiao Xu. 2015. Region-based structure line detection for cartoons. Computational Visual Media 1, 1 (2015), 69–78. Google ScholarCross Ref
21. Yusuke Matsui, Kota Ito, Yuji Aramaki, Toshihiko Yamasaki, and Kiyoharu Aizawa. 2015. Sketch-based Manga Retrieval using Manga109 Dataset. CoRR abs/1510.04389 (2015).Google Scholar
22. Hyeonwoo Noh, Seunghoon Hong, and Bohyung Han. 2015. Learning Deconvolution Network for Semantic Segmentation. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7–13, 2015. 1520–1528. Google ScholarDigital Library
23. Yingge Qu, Tien-Tsin Wong, and Pheng-Ann Heng. 2006. Manga colorization. ACM Transactions on Graphics (TOG) 25, 3 (2006), 1214–1220. Google ScholarDigital Library
24. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-Net: Convolutional Networks for Biomedical Image Segmentation. In Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015 – 18th International Conference Munich, Germany, October 5 – 9, 2015, Proceedings, Part III. 234–241. Google ScholarCross Ref
25. Wei Shen, Xinggang Wang, Yan Wang, Xiang Bai, and Zhijiang Zhang. 2015. Deep-Contour: A deep convolutional feature learned by positive-sharing loss for contour detection. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7–12, 2015. 3982–3991.Google ScholarCross Ref
26. Edgar Simo-Serra, Satoshi Iizuka, Kazuma Sasaki, and Hiroshi Ishikawa. 2016. Learning to simplify: fully convolutional networks for rough sketch cleanup. ACM Trans. Graph. 35, 4 (2016), 121. Google ScholarDigital Library
27. Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. CoRR abs/1409.1556 (2014).Google Scholar
28. Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin A. Riedmiller. 2014. Striving for Simplicity: The All Convolutional Net. CoRR abs/1412.6806 (2014).Google Scholar
29. Daniel Sýkora, Jan Buriánek, and Jiří Žára. 2004. Unsupervised colorization of black-and-white cartoons. In International Symposium on Non-Photorealistic Animation and Rendering. 121–127. Google ScholarDigital Library
30. Carlo Tomasi and Roberto Manduchi. 1998. Bilateral Filtering for Gray and Color Images. In ICCV. 839–846. Google ScholarCross Ref
31. Luminita A. Vese and Stanley Osher. 2003. Modeling Textures with Total Variation Minimization and Oscillating Patterns in Image Processing. J. Sci. Comput. 19, 1–3 (2003), 553–572.Google ScholarDigital Library
32. Saining Xie and Zhuowen Tu. 2015. Holistically-Nested Edge Detection. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7–13, 2015. 1395–1403. Google ScholarDigital Library
33. Li Xu, Cewu Lu, Yi Xu, and Jiaya Jia. 2011. Image smoothing via L0 gradient minimization. ACM Trans. Graph. 30, 6 (2011), 174. Google ScholarDigital Library
34. Li Xu, Qiong Yan, Yang Xia, and Jiaya Jia. 2012. Structure extraction from texture via relative total variation. ACM Trans. Graph. 31, 6 (2012), 139. Google ScholarDigital Library
35. Chih-Yuan Yao, Shih-Hsuan Hung, Guo-Wei Li, I-Yu Chen, Reza Adhitya, and Yu-Chi Lai. 2017. Manga Vectorization and Manipulation with Procedural Simple Screentone. IEEE Trans. Vis. Comput. Graph. 23, 2 (2017), 1070–1084. Google ScholarDigital Library
36. Song-Hai Zhang, Tao Chen, Yi-Fei Zhang, Shi-Min Hu, and Ralph R Martin. 2009. Vectorizing cartoon animations. IEEE Transactions on Visualization and Computer Graphics 15, 4 (2009), 618–629.Google ScholarDigital Library
37. Hang Zhao, Orazio Gallo, Iuri Frosio, and Jan Kautz. 2016. Loss Functions for Neural Networks for Image Processing. CoRR abs/1511.08861 (2016).Google Scholar
