“Deep unsupervised pixelization”
Conference:
Type(s):
Title:
- Deep unsupervised pixelization
Session/Category Title: Image processing
Presenter(s)/Author(s):
Moderator(s):
Abstract:
In this paper, we present a novel unsupervised learning method for pixelization. Due to the difficulty in creating pixel art, preparing the paired training data for supervised learning is impractical. Instead, we propose an unsupervised learning framework to circumvent such difficulty. We leverage the dual nature of the pixelization and depixelization, and model these two tasks in the same network in a bi-directional manner with the input itself as training supervision. These two tasks are modeled as a cascaded network which consists of three stages for different purposes. GridNet transfers the input image into multi-scale grid-structured images with different aliasing effects. PixelNet associated with GridNet to synthesize pixel arts with sharp edges and perceptually optimal local structures. DepixelNet connects the previous network and aims to recover the pixelized result to the original image. For the sake of unsupervised learning, the mirror loss is proposed to hold the reversibility of feature representations in the process. In addition, adversarial, L1, and gradient losses are involved in the network to obtain pixel arts by retaining color correctness and smoothness. We show that our technique can synthesize crisper and perceptually more appropriate pixel arts than state-of-the-art image downscaling methods. We evaluate the proposed method with extensive experiments on many images. The proposed method outperforms state-of-the-art methods in terms of visual quality and user preference.
References:
1. Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk. 2012. SLIC superpixels compared to state-of-the-art superpixel methods. IEEE transactions on pattern analysis and machine intelligence 34, 11 (2012), 2274–2282. Google ScholarDigital Library
2. Qifeng Chen and Vladlen Koltun. 2017. Photographic image synthesis with cascaded refinement networks. In The IEEE International Conference on Computer Vision (ICCV), Vol. 1.Google ScholarCross Ref
3. Mark AZ Dippé and Erling Henry Wold. 1985. Antialiasing through stochastic sampling. ACM Siggraph Computer Graphics 19, 3 (1985), 69–78. Google ScholarDigital Library
4. Timothy Gerstner, Doug DeCarlo, Marc Alexa, Adam Finkelstein, Yotam Gingold, and Andrew Nealen. 2012. Pixelated image abstraction. In Proceedings of the Symposium on Non-Photorealistic Animation and Rendering. Eurographics Association, 29–36. Google ScholarDigital Library
5. Adele Goldberg and Robert Flegal. 1982. ACM president’s letter: Pixel Art. Commun. ACM 25, 12 (1982), 861–862. Google ScholarDigital Library
6. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in neural information processing systems. 2672–2680. Google ScholarDigital Library
7. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770–778.Google ScholarCross Ref
8. Tiffany C Inglis and Craig S Kaplan. 2012. Pixelating vector line art. In Proceedings of the Symposium on Non-Photorealistic Animation and Rendering. Eurographics Association, 21–28. Google ScholarDigital Library
9. Tiffany C Inglis, Daniel Vogel, and Craig S Kaplan. 2013. Rasterizing and antialiasing vector line art in the pixel art style. In proceedings of the symposium on non-photorealistic animation and rendering. ACM, 25–32. Google ScholarDigital Library
10. Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017. Image-to-image translation with conditional adversarial networks. arXiv preprint (2017).Google Scholar
11. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).Google Scholar
12. Johannes Kopf and Dani Lischinski. 2011. Depixelizing pixel art. In ACM Transactions on graphics (TOG), Vol. 30. ACM, 99. Google ScholarDigital Library
13. Johannes Kopf, Ariel Shamir, and Pieter Peers. 2013. Content-adaptive image downscaling. ACM Transactions on Graphics (TOG) 32, 6 (2013), 173. Google ScholarDigital Library
14. Ming-Hsun Kuo, Yong-Liang Yang, and Hung-Kuo Chu. 2016. Feature-Aware Pixel Art Animation. In Computer Graphics Forum, Vol. 35. Wiley Online Library, 411–420. Google ScholarDigital Library
15. Jonathan Long, Evan Shelhamer, and Trevor Darrell. 2015. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition. 3431–3440.Google ScholarCross Ref
16. Mehdi Mirza and Simon Osindero. 2014. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 (2014).Google Scholar
17. Augustus Odena, Christopher Olah, and Jonathon Shlens. 2016. Conditional image synthesis with auxiliary classifier gans. arXiv preprint arXiv:1610.09585 (2016).Google ScholarDigital Library
18. A Cengiz Öztireli and Markus Gross. 2015. Perceptually based downscaling of images. ACM Transactions on Graphics (TOG) 34, 4 (2015), 77. Google ScholarDigital Library
19. Claude Elwood Shannon. 1949. Communication in the presence of noise. Proceedings of the IRE 37, 1 (1949), 10–21.Google Scholar
20. Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Josh Susskind, Wenda Wang, and Russ Webb. 2017. Learning from simulated and unsupervised images through adversarial training. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vol. 3. 6.Google ScholarCross Ref
21. Nicolas Weber, Michael Waechter, Sandra C Amend, Stefan Guthe, and Michael Goesele. 2016. Rapid, detail-preserving image downscaling. ACM Transactions on Graphics (TOG) 35, 6 (2016), 205. Google ScholarDigital Library
22. Saining Xie and Zhuowen Tu. 2015. Holistically-nested edge detection. In Proceedings of the IEEE international conference on computer vision. 1395–1403. Google ScholarDigital Library
23. Zili Yi, Hao Zhang, Ping Tan, and Minglun Gong. 2017. Dualgan: Unsupervised dual learning for image-to-image translation. arXiv preprint (2017).Google Scholar
24. Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. 2017. Unpaired image-to-image translation using cycle-consistent adversarial networks. arXiv preprint arXiv:1703.10593 (2017).Google Scholar
