“Deep context-aware descreening and rescreening of halftone images” by Kim and Park
Conference:
Type:
Entry Number: 48
Title:
- Deep context-aware descreening and rescreening of halftone images
Session/Category Title: Computational Photography
Presenter(s)/Author(s):
Moderator(s):
Abstract:
A fully automatic method for descreening halftone images is presented based on convolutional neural networks with end-to-end learning. Incorporating context level information, the proposed method not only removes halftone artifacts but also synthesizes the fine details lost during halftone. The method consists of two main stages. In the first stage, intrinsic features of the scene are extracted, the low-frequency reconstruction of the image is estimated, and halftone patterns are removed. For the intrinsic features, the edges and object-categories are estimated and fed to the next stage as strong visual and contextual cues. In the second stage, fine details are synthesized on top of the low-frequency output based on an adversarial generative model. In addition, the novel problem of rescreening is addressed, where a natural input image is halftoned so as to be similar to a separately given reference halftone image. To this end, a two-stage convolutional neural network is also presented. Both networks are trained with millions of before-and-after example image pairs of various halftone styles. Qualitative and quantitative evaluations are provided, which demonstrates the effectiveness of the proposed methods.
References:
1. Gedas Bertasius, Jianbo Shi, and Lorenzo Torresani. 2015. DeepEdge: A Multi-Scale Bifurcated Deep Network for Top-Down Contour Detection. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
2. John Canny. 1986. A Computational Approach to Edge Detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 8, 6 (Nov 1986), 679–698. Google ScholarDigital Library
3. Hojin Cho, Hyunjoon Lee, Henry Kang, and Seungyong Lee. 2014. Bilateral Texture Filtering. ACM Transactions on Graphics (Proceedings of SIGGRAPH 2014) 33, 4 (2014), 128:1–128:8. Google ScholarDigital Library
4. Fernando de Goes, Katherine Breeden, Victor Ostromoukhov, and Mathieu Desbrun. 2012. Blue Noise Through Optimal Transport. ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia 2012) 31, 6 (2012), 171:1–171:11. Google ScholarDigital Library
5. Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. 2016. Image Style Transfer Using Convolutional Neural Networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2414–2423.Google Scholar
6. Leon A. Gatys, Alexander S. Ecker, Matthias Bethge, Aaron Hertzmann, and Eli Shechtman. 2017. Controlling Perceptual Factors in Neural Style Transfer. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 3730–3738.Google Scholar
7. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative Adversarial Nets. In Advances in Neural Information Processing Systems 27. 2672–2680. Google ScholarDigital Library
8. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. In IEEE International Conference on Computer Vision (ICCV). 1026–1034. Google ScholarDigital Library
9. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
10. Aaron Hertzmann, Charles E. Jacobs, Nuria Oliver, Brian Curless, and David H. Salesin. 2001. Image Analogies. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’01). ACM, New York, NY, USA, 327–340. Google ScholarDigital Library
11. Xianxu Hou and Guoping Qiu. 2017. Image Companding and Inverse Halftoning using Deep Convolutional Neural Networks. CoRR abs/1707.00116 (2017). arXiv:1707.00116 http://arxiv.org/abs/1707.00116Google Scholar
12. Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. 2016. Let there be Color!: Joint End-to-end Learning of Global and Local Image Priors for Automatic Image Colorization with Simultaneous Classification. ACM Transactions on Graphics (Proc. of SIGGRAPH 2016) 35, 4 (2016), 110:1–110:11. Google ScholarDigital Library
13. Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei Efros. 2017. Image-to-Image Translation with Conditional Adversarial Networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
14. Henry R. Kang. 1999. Digital Color Halftoning. SPIE Press. Google ScholarDigital Library
15. Levent Karacan, Erkut Erdem, and Aykut Erdem. 2013. Structure-preserving Image Smoothing via Region Covariances. ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia 2013) 32, 6 (2013), 176:1–176:11. Google ScholarDigital Library
16. Yeong-Taeg Kim, Gonzalo R. Arce, and Nikolai Grabowski. 1995. Inverse Halftoning Using Binary Permutation Filters. IEEE Transactions on Image Processing 4, 9 (1995), 1296–1311. Google ScholarDigital Library
17. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. CoRR abs/1412.6980 (2014).Google Scholar
18. Johannes Kopf and Dani Lischinski. 2012. Digital Reconstruction of Halftoned Color Comics. ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia 2012) 31, 6 (2012). Google ScholarDigital Library
19. Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew P. Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi. 2017. Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 105–114.Google ScholarCross Ref
20. Chengze Li, Xueting Liu, and Tien-Tsin Wong. 2017. Deep Extraction of Manga Structural Lines. ACM Trans. Graph. 36, 4, Article 117 (July 2017), 12 pages. Google ScholarDigital Library
21. Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung M. Lee. 2017. Enhanced Deep Residual Networks for Single Image Super-Resolution. 2nd NTIRE: New Trends in Image Restoration and Enhancement workshop and challenge on image super-resolution in conjunction with CVPR abs/1707.02921 (2017).Google Scholar
22. Jiebo Luo, Ricardo de Queiroz, and Zhigang Fan. 1998. A Robust Technique for Image Descreening Based on the Wavelet Transform. IEEE Transactions on Signal Processing 46, 4 (1998), 1179–1184. Google ScholarDigital Library
23. Xudong Mao, Qing Li, Haoran Xie, Raymond Y.K. Lau, Zhen Wang, and Stephen Paul Smolley. 2017. Least squares generative adversarial networks. In IEEE International Conference on Computer Vision (ICCV).Google ScholarCross Ref
24. Victor Ostromoukhov. 1999. Digital Facial Engraving. In SIGGRAPH 99. Google ScholarDigital Library
25. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch. (2017).Google Scholar
26. Deepak Pathak, Philipp Krähenbühl, Jeff Donahue, Trevor Darrell, and Alexei Efros. 2016. Context Encoders: Feature Learning by Inpainting. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google ScholarCross Ref
27. O. Ronneberger, P.Fischer, and T. Brox. 2015. U-Net: Convolutional Networks for Biomedical Image Segmentation. In Medical Image Computing and Computer-Assisted Intervention (MICCAI) (LNCS), Vol. 9351. Springer, 234–241. (available on arXiv:1505.04597 {cs.CV}).Google Scholar
28. M. S. M. Sajjadi, B. Schölkopf, and M. Hirsch. 2017. EnhanceNet: Single Image Super-Resolution through Automated Texture Synthesis, In IEEE International Conference on Computer Vision (ICCV). arXiv:1612.07919 (2017), 4491–4500.Google Scholar
29. Wei Shen, Xinggang Wang, Yan Wang, Xiang Bai, and Zhijiang Zhang. 2015. DeepContour: A Deep Convolutional Feature Learned by Positive-Sharing Loss for Contour Detection. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
30. Yu-Wen Shou and Chin-Teng Lin. 2010. Hardware-Friendly Descreening. IEEE Transactions on Circuits and Systems 19, 3 (2010), 2287–2299. Google ScholarDigital Library
31. Hasib Siddiqui and Charles A. Bouman. 2007. Training-Based Descreening. IEEE Transactions on Image Processing 16, 3 (2007), 789–802. Google ScholarDigital Library
32. K. Simonyan and A. Zisserman. 2014. Very Deep Convolutional Networks for Large-Scale Image Recognition. CoRR abs/1409.1556 (2014).Google Scholar
33. Aäron Van Den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. 2016. Pixel Recurrent Neural Networks. In Proceedings of the 33rd International Conference on International Conference on Machine Learning – Volume 48 (ICML’16). 1747–1756. Google ScholarDigital Library
34. Ping Wah Wong. 1995. Inverse Halftoning and Kernel Estimation for Error Diffusion. IEEE Transactions on Image Processing 4, 4 (1995), 486–498. Google ScholarDigital Library
35. Saining Xie and Zhuowen Tu. 2015. Holistically-Nested Edge Detection. In IEEE International Conference on Computer Vision (ICCV). Google ScholarDigital Library
36. Zixiang Xiong, Michael T. Orchard, and Kannan Ramchandran. 1999. Inverse Halfoning Using Wavelets. IEEE Transactions on Image Processing 8, 10 (1999), 1479–1483. Google ScholarDigital Library
37. Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. 2017a. Places: A 10 million Image Database for Scene Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence (2017).Google Scholar
38. Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. 2017b. Scene Parsing through ADE20K Dataset. In IEEE Conference on Computer Vision and Pattern Recognition.Google ScholarCross Ref
39. Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. 2017a. Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks. arXiv preprint arXiv:1703.10593 (2017).Google Scholar
40. Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A Efros, Oliver Wang, and Eli Shechtman. 2017b. Toward Multimodal Image-to-Image Translation. In Advances in Neural Information Processing Systems 30.Google Scholar
