“Disentangled Image Colorization via Global Anchors” by Xia, Hu, Wong and Wang
Conference:
Type(s):
Title:
- Disentangled Image Colorization via Global Anchors
Session/Category Title: Styilzation and Colorization
Presenter(s)/Author(s):
Abstract:
Colorization is multimodal by nature and challenges existing frameworks to achieve colorful and structurally consistent results. Even the sophisticated autoregressive model struggles to maintain long-distance color consistency due to the fragility of sequential dependence. To overcome this challenge, we propose a novel colorization framework that disentangles color multimodality and structure consistency through global color anchors, so that both aspects could be learned effectively. Our key insight is that several carefully located anchors could approximately represent the color distribution of an image, and conditioned on the anchor colors, we can predict the image color in a deterministic manner by utilizing internal correlation. To this end, we construct a colorization model with dual branches, where the color modeler predicts the color distribution for anchor color representation, and the color generator predicts the pixel colors by referring the sampled anchor colors. Importantly, the anchors are located under two principles: color independence and global coverage, which is realized with clustering analysis on the deep color features. To simplify the computation, we creatively adopt soft superpixel segmentation to reduce the image primitives, which still nicely reserves the reversibility to pixel-wise representation. Extensive experiments show that our method achieves notable superiority over various mainstream frameworks in perceptual quality. Thanks to anchor-based color representation, our model has the flexibility to support diverse and controllable colorization as well.
References:
1. Rameen Abdal, Yipeng Qin, and Peter Wonka. 2019. Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?. In IEEE International Conference on Computer Vision (ICCV).
2. Jason Antic. 2019. DeOldify: A open-source project for colorizing old images (and video).
3. Aurélie Bugeau, Vinh-Thong Ta, and Nicolas Papadakis. 2014. Variational Exemplar-Based Image Colorization. IEEE Trans. Image Process. (TIP) 23, 1 (2014), 298–307.
4. Holger Caesar, Jasper R. R. Uijlings, and Vittorio Ferrari. 2018. COCO-Stuff: Thing and Stuff Classes in Context. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
5. Huiwen Chang, Ohad Fried, Yiming Liu, Stephen DiVerdi, and Adam Finkelstein. 2015. Palette-based photo recoloring. ACM Trans. Graph. (TOG) 34, 4 (2015), 139:1–139:11.
6. Zezhou Cheng, Qingxiong Yang, and Bin Sheng. 2015. Deep Colorization. In IEEE International Conference on Computer Vision (ICCV).
7. Alex Yong-Sang Chia, Shaojie Zhuo, Raj Kumar Gupta, Yu-Wing Tai, Siu-Yeung Cho, Ping Tan, and Stephen Lin. 2011. Semantic colorization with internet images. ACM Trans. Graph. (TOG) 30, 6 (2011), 1–8.
8. Wonwoong Cho, Hyojin Bahng, David Keetae Park, Seungjoo Yoo, Ziming Wu, Xiaojuan Ma, and Jaegul Choo. 2018. Text2Colors: Guiding Image Colorization through Text-Driven Palette Generation. In European Conference on Computer Vision (ECCV).
9. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. ImageNet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
10. Aditya Deshpande, Jiajun Lu, Mao-Chuang Yeh, Min Jin Chong, and David A. Forsyth. 2017. Learning Diverse Image Colorization. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
11. Aditya Deshpande, Jason Rock, and David A. Forsyth. 2015. Learning Large-Scale Automatic Image Colorization. In IEEE International Conference on Computer Vision (ICCV).
12. Jinjin Gu, Yujun Shen, and Bolei Zhou. 2020. Image Processing Using Multi-Code GAN Prior. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
13. Sergio Guadarrama, Ryan Dahl, David Bieber, Jonathon Shlens, Mohammad Norouzi, and Kevin Murphy. 2017. PixColor: Pixel Recursive Colorization. In British Machine Vision Conference 2017 (BMVC).
14. David Hasler and Sabine Süsstrunk. 2003. Measuring colorfulness in natural images. In Human Vision and Electronic Imaging VIII.
15. Mingming He, Dongdong Chen, Jing Liao, Pedro V. Sander, and Lu Yuan. 2018. Deep exemplar-based colorization. ACM Trans. Graph. (TOG) 37, 4 (2018), 47:1–47:16.
16. Mingming He, Jing Liao, Dongdong Chen, Lu Yuan, and Pedro V. Sander. 2019. Progressive Color Transfer With Dense Semantic Correspondences. ACM Trans. Graph. (TOG) 38, 2 (2019), 13:1–13:18.
17. Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In Annual Conference on Neural Information Processing Systems (NeurIPS).
18. Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. 2016. Let there be color! Joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification. ACM Trans. Graph. (TOG) 35, 4 (2016), 1–11.
19. Revital Ironi, Daniel Cohen-Or, and Dani Lischinski. 2005. Colorization by Example. In Eurographics Symposium on Rendering Techniques.
20. Eungyeup Kim, Sanghyeon Lee, Jeonghoon Park, Somi Choi, Choonghyun Seo, and Jaegul Choo. 2021. Deep Edge-Aware Interactive Colorization against Color-Bleeding Effects. In IEEE International Conference on Computer Vision (ICCV).
21. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint:1511.06349 (2014).
22. Manoj Kumar, DirkWeissenborn, and Nal Kalchbrenner. 2021. Colorization Transformer. In International Conference on Learning Representations (ICLR).
23. Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. 2016. Learning Representations for Automatic Colorization. In European Conference on Computer Vision (ECCV).
24. Anat Levin, Dani Lischinski, and Yair Weiss. 2004. Colorization using optimization. ACM Trans. Graph. (TOG) 23, 3 (2004), 689–694.
25. Bo Li, Fuchen Zhao, Zhuo Su, Xiangguo Liang, Yu-Kun Lai, and Paul L. Rosin. 2017. Example-Based Image Colorization Using Locality Consistent Sparse Representation. IEEE Trans. Image Process. (TIP) 26, 11 (2017), 5188–5202.
26. Xuan Luo, Yanmeng Kong, Jason Lawrence, Ricardo Martin-Brualla, and Steven M. Seitz. 2020. KeystoneDepth: History in 3D. In International Conference on 3D Vision (3DV). https://keystonedepth.cs.washington.edu
27. Safa Messaoud, David A. Forsyth, and Alexander G. Schwing. 2018. Structural Consistency and Controllability for Diverse Colorization. In European Conference on Computer Vision (ECCV).
28. Yingge Qu, Tien-Tsin Wong, and Pheng-Ann Heng. 2006. Manga colorization. ACM Trans. Graph. (TOG) 25, 3 (2006), 1214–1220.
29. Tim Salimans, Ian J. Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. 2016. Improved Techniques for Training GANs. In Annual Conference on Neural Information Processing Systems (NeurIPS).
30. Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. In International Conference on Learning Representations (ICLR).
31. Jheng-Wei Su, Hung-Kuo Chu, and Jia-Bin Huang. 2020. Instance-Aware Image Colorization. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
32. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Annual Conference on Neural Information Processing Systems (NeurIPS).
33. Patricia Vitoria, Lara Raad, and Coloma Ballester. 2020. ChromaGAN: Adversarial Picture Colorization with Semantic Class Distribution. In IEEE Winter Conference on Applications of Computer Vision (WACV).
34. Tomihisa Welsh, Michael Ashikhmin, and Klaus Mueller. 2002. Transferring color to greyscale images. ACM Trans. Graph. (TOG) 21, 3 (2002), 277–280.
35. Yanze Wu, Xintao Wang, Yu Li, Honglun Zhang, Xun Zhao, and Ying Shan. 2021. Towards Vivid and Diverse Image Colorization with Generative Color Prior. In IEEE International Conference on Computer Vision (ICCV).
36. Fengting Yang, Qian Sun, Hailin Jin, and Zihan Zhou. 2020. Superpixel Segmentation With Fully Convolutional Networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
37. Liron Yatziv and Guillermo Sapiro. 2006. Fast image and video colorization using chrominance blending. IEEE Trans. Image Process. (TIP) 15, 5 (2006), 1120–1129.
38. Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. 2019. Self-Attention Generative Adversarial Networks. In International Conference on Machine Learning (ICML).
39. Richard Zhang, Phillip Isola, and Alexei A. Efros. 2016. Colorful Image Colorization. In European Conference on Computer Vision (ECCV).
40. Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. 2018. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
41. Richard Zhang, Jun-Yan Zhu, Phillip Isola, Xinyang Geng, Angela S. Lin, Tianhe Yu, and Alexei A. Efros. 2017. Real-time user-guided image colorization with learned deep priors. ACM Trans. Graph. (TOG) 36, 4 (2017), 119:1–119:11.
42. Jiaojiao Zhao, Jungong Han, Ling Shao, and Cees G. M. Snoek. 2020. Pixelated Semantic Colorization. Int. J. Comput. Vis. (IJCV) 128, 4 (2020), 818–834.
43. Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl. 2020. Tracking Objects as Points. In European Conference on Computer Vision (ECCV).
