“Deep exemplar-based colorization” by He, Chen, Liao, Sander and Yuan
Conference:
Type(s):
Entry Number: 47
Title:
- Deep exemplar-based colorization
Session/Category Title: Computational Photography
Presenter(s)/Author(s):
Moderator(s):
Abstract:
We propose the first deep learning approach for exemplar-based local colorization. Given a reference color image, our convolutional neural network directly maps a grayscale image to an output colorized image. Rather than using hand-crafted rules as in traditional exemplar-based methods, our end-to-end colorization network learns how to select, propagate, and predict colors from the large-scale data. The approach performs robustly and generalizes well even when using reference images that are unrelated to the input grayscale image. More importantly, as opposed to other learning-based colorization methods, our network allows the user to achieve customizable results by simply feeding different references. In order to further reduce manual effort in selecting the references, the system automatically recommends references with our proposed image retrieval algorithm, which considers both semantic and luminance information. The colorization can be performed fully automatically by simply picking the top reference suggestion. Our approach is validated through a user study and favorable quantitative comparisons to the-state-of-the-art methods. Furthermore, our approach can be naturally extended to video colorization. Our code and models are freely available for public use.
References:
1. Artem Babenko and Victor Lempitsky. 2015. Aggregating local deep features for image retrieval. In Proc. ICCV. 1269–1277. Google ScholarDigital Library
2. Artem Babenko, Anton Slesarev, Alexandr Chigorin, and Victor Lempitsky. 2014. Neural codes for image retrieval. In Proc. ECCV. Springer, 584–599.Google Scholar
3. Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. 2015. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. arXiv preprint arXiv:1511.00561 (2015).Google Scholar
4. Nicolas Bonneel, James Tompkin, Kalyan Sunkavalli, Deqing Sun, Sylvain Paris, and Hanspeter Pfister. 2015. Blind video temporal consistency. ACM Trans. Graph. (Proc. of SIGGRAPH Asia) 34, 6 (2015), 196. Google ScholarDigital Library
5. Aurélie Bugeau and Vinh-Thong Ta. 2012. Patch-based image colorization. In Pattern Recognition (ICPR), 2012 21st International Conference on. IEEE, 3058–3061.Google Scholar
6. Aurélie Bugeau, Vinh-Thong Ta, and Nicolas Papadakis. 2014. Variational exemplar-based image colorization. IEEE Trans. on Image Processing 23, 1 (2014), 298–307. Google ScholarDigital Library
7. Guillaume Charpiat, Matthias Hofmann, and Bernhard Schölkopf. 2008. Automatic image colorization via multimodal predictions. (2008), 126–139. Google ScholarDigital Library
8. Dongdong Chen, Jing Liao, Lu Yuan, Nenghai Yu, and Gang Hua. 2017a. Coherent online video style transfer. In Proc. ICCV.Google ScholarCross Ref
9. Dongdong Chen, Lu Yuan, Jing Liao, Nenghai Yu, and Gang Hua. 2017b. Stylebank: An explicit representation for neural image style transfer. In Proc. CVPR.Google ScholarCross Ref
10. Dongdong Chen, Lu Yuan, Jing Liao, Nenghai Yu, and Gang Hua. 2018. Stereoscopic Neural Style Transfer. In Proc. CVPR.Google ScholarCross Ref
11. Qifeng Chen and Vladlen Koltun. 2017. Photographic image synthesis with cascaded refinement networks. In Proc. ICCV, Vol. 1.Google ScholarCross Ref
12. Zezhou Cheng, Qingxiong Yang, and Bin Sheng. 2015. Deep colorization. In Proc. ICCV. 415–423. Google ScholarDigital Library
13. Alex Yong-Sang Chia, Shaojie Zhuo, Raj Kumar Gupta, Yu-Wing Tai, Siu-Yeung Cho, Ping Tan, and Stephen Lin. 2011. Semantic colorization with internet images. ACM Trans. Graph. (Proc. of SIGGRAPH Asia) 30, 6 (2011), 156. Google ScholarDigital Library
14. Aditya Deshpande, Jason Rock, and David Forsyth. 2015. Learning large-scale automatic image colorization. In Proc. ICCV. 567–575. Google ScholarDigital Library
15. Yunchao Gong, Liwei Wang, Ruiqi Guo, and Svetlana Lazebnik. 2014. Multi-scale orderless pooling of deep convolutional activation features. In Proc. ECCV. Springer, 392–407.Google Scholar
16. Raj Kumar Gupta, Alex Yong-Sang Chia, Deepu Rajan, Ee Sin Ng, and Huang Zhiyong. 2012. Image colorization using similar images. In Proc. of the 20th ACM international conference on Multimedia. ACM, 369–378. Google ScholarDigital Library
17. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Deep Residual Learning for Image Recognition. arXiv preprint arXiv:1512.03385 (2015).Google Scholar
18. Mingming He, Jing Liao, Lu Yuan, and Pedro V Sander. 2017. Neural Color Transfer between Images. arXiv preprint arXiv:1710.00756 (2017).Google Scholar
19. Yi-Chin Huang, Yi-Shin Tung, Jun-Cheng Chen, Sung-Wen Wang, and Ja-Ling Wu. 2005. An adaptive edge detection based colorization algorithm and its applications. In Proc. of the 13th annual ACM international conference on Multimedia. ACM, 351–354. Google ScholarDigital Library
20. Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. 2016. Let there be color!: joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification. ACM Trans. Graph. (Proc. of SIGGRAPH) 35,4 (2016), 110. Google ScholarDigital Library
21. Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning. 448–456. Google ScholarDigital Library
22. Revital Irony, Daniel Cohen-Or, and Dani Lischinski. 2005. Colorization by Example. In Rendering Techniques. 201–210. Google ScholarDigital Library
23. Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017. Image-to-image translation with conditional adversarial networks. In Proc. CVPR.Google ScholarCross Ref
24. Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016. Perceptual losses for real-time style transfer and super-resolution. In Proc. ECCV. Springer, 694–711.Google Scholar
25. Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).Google Scholar
26. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems. 1097–1105. Google ScholarDigital Library
27. Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. 2016. Learning representations for automatic colorization. In Proc. ECCV. 577–593.Google ScholarCross Ref
28. Anat Levin, Dani Lischinski, and Yair Weiss. 2004. Colorization using optimization. ACM Trans. Graph. (Proc. of SIGGRAPH) 23, 3 (2004), 689–694. Google ScholarDigital Library
29. Jing Liao, Yuan Yao, Lu Yuan, Gang Hua, and Sing Bing Kang. 2017. Visual Attribute Transfer through Deep Image Analogy. arXiv preprint arXiv:1705.01088 36, 4 (2017), 120. Google ScholarDigital Library
30. Ce Liu, Jenny Yuen, and Antonio Torralba. 2011. Sift flow: Dense correspondence across scenes and its applications. IEEE Trans. Pattern Anal. Mach. Intell. 33, 5 (2011), 978–994. Google ScholarDigital Library
31. Xiaopei Liu, Liang Wan, Yingge Qu, Tien-Tsin Wong, Stephen Lin, Chi-Sing Leung, and Pheng-Ann Heng. 2008. Intrinsic colorization. ACM Trans. Graph. (Proc. of SIGGRAPH Asia) 27, 5 (2008), 152. Google ScholarDigital Library
32. Qing Luan, Fang Wen, Daniel Cohen-Or, Lin Liang, Ying-Qing Xu, and Heung-Yeung Shum. 2007. Natural image colorization. In Proc. of the 18th Eurographics conference on Rendering Techniques. Eurographics Association, 309–320. Google ScholarDigital Library
33. Yingge Qu, Tien-Tsin Wong, and Pheng-Ann Heng. 2006. Manga colorization. ACM Trans. Graph. (Proc. of SIGGRAPH Asia) 25, 3 (2006), 1214–1220. Google ScholarDigital Library
34. Alec Radford, Luke Metz, and Soumith Chintala. 2015. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434 (2015).Google Scholar
35. Ali Sharif Razavian, Josephine Sullivan, Stefan Carlsson, and Atsuto Maki. 2016. A baseline for visual instance retrieval with deep convolutional networks. arXiv preprint arXiv:1412.6574 (2016).Google Scholar
36. Erik Reinhard, Michael Adhikhmin, Bruce Gooch, and Peter Shirley. 2001. Color transfer between images. IEEE Computer graphics and applications 21, 5 (2001), 34–41. Google ScholarDigital Library
37. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, and others. 2015. Imagenet large scale visual recognition challenge. International Journal of Computer Vision 115, 3 (2015), 211–252. Google ScholarDigital Library
38. Mehdi SM Sajjadi, Bernhard Scholkopf, and Michael Hirsch. 2017. Enhancenet: Single image super-resolution through automated texture synthesis. In Proc. CVPR. 4491–4500.Google ScholarCross Ref
39. Patsorn Sangkloy, Jingwan Lu, Chen Fang, Fisher Yu, and James Hays. 2016. Scribbler: Controlling deep image synthesis with sketch and color. In Proc. CVPR.Google Scholar
40. Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015. Going Deeper with Convolutions. In Proc. CVPR.Google ScholarCross Ref
41. Yu-Wing Tai, Jiaya Jia, and Chi-Keung Tang. 2005. Local color transfer via probabilistic segmentation by expectation-maximization. In Proc. CVPR, Vol. 1. IEEE, 747–754. Google ScholarDigital Library
42. Engin Tola, Vincent Lepetit, and Pascal Fua. 2010. Daisy: An efficient dense descriptor applied to wide-baseline stereo. IEEE Trans. Pattern Anal. Mach. Intell. 32, 5 (2010), 815–830. Google ScholarDigital Library
43. Giorgos Tolias, Ronan Sicre, and Hervé Jégou. 2015. Particular object retrieval with integral max-pooling of CNN activations. arXiv preprint arXiv:1511.05879 (2015).Google Scholar
44. Philippe Weinzaepfel, Jerome Revaud, Zaid Harchaoui, and Cordelia Schmid. 2013. DeepFlow: Large displacement optical flow with deep matching. In Proc. ICCV. 1385–1392. Google ScholarDigital Library
45. Tomihisa Welsh, Michael Ashikhmin, and Klaus Mueller. 2002. Transferring color to greyscale images. ACM Trans. Graph. (Proc. of SIGGRAPH Asia) 21, 3 (2002), 277–280. Google ScholarDigital Library
46. Liron Yatziv and Guillermo Sapiro. 2006. Fast image and video colorization using chrominance blending. IEEE Trans. on Image Processing 15, 5 (2006), 1120–1129. Google ScholarDigital Library
47. Fisher Yu and Vladlen Koltun. 2015. Multi-scale context aggregation by dilated convolutions. arXiv preprint arXiv:1511.07122 (2015).Google Scholar
48. Richard Zhang, Phillip Isola, and Alexei A Efros. 2016. Colorful image colorization. In Proc. ECCV. 649–666.Google ScholarCross Ref
49. Richard Zhang, Jun-Yan Zhu, Phillip Isola, Xinyang Geng, Angela S Lin, Tianhe Yu, and Alexei A Efros. 2017. Real-time user-guided image colorization with learned deep priors. ACM Trans. Graph. (Proc. of SIGGRAPH) 36, 4 (2017), 119. Google ScholarDigital Library
