“Visual attribute transfer through deep image analogy” by Liao, Yao, Yuan, Hua and Kang
Conference:
Type:
Title:
- Visual attribute transfer through deep image analogy
Session/Category Title: Deep Image Processing
Presenter(s)/Author(s):
Moderator(s):
Abstract:
We propose a new technique for visual attribute transfer across images that may have very different appearance but have perceptually similar semantic structure. By visual attribute transfer, we mean transfer of visual information (such as color, tone, texture, and style) from one image to another. For example, one image could be that of a painting or a sketch while the other is a photo of a real scene, and both depict the same type of scene.Our technique finds semantically-meaningful dense correspondences between two input images. To accomplish this, it adapts the notion of “image analogy” [Hertzmann et al. 2001] with features extracted from a Deep Convolutional Neutral Network for matching; we call our technique deep image analogy. A coarse-to-fine strategy is used to compute the nearest-neighbor field for generating the results. We validate the effectiveness of our proposed method in a variety of cases, including style/texture transfer, color/style swap, sketch/painting to photo, and time lapse.
References:
1. Mike Tyka Alexander Mordvintsev, Christopher Olah. 2015. Inceptionism: Going Deeper into Neural Networks. (2015). https://research.googleblog.com/2015/06/inceptionism-going-deeper-into-neural.htmlGoogle Scholar
2. Xiaobo An and Fabio Pellacini. 2010. User-Controllable Color Transfer. Computer Graphics Forum (2010).Google Scholar
3. Michael Ashikhmin. 2003. Fast texture transfer. IEEE Comput. Graph. and Appl. 23, 4 (2003). Google ScholarDigital Library
4. Simon Baker, Daniel Scharstein, JP Lewis, Stefan Roth, Michael J Black, and Richard Szeliski. 2011. A database and evaluation methodology for optical flow. International Journal of Computer Vision 92, 1 (2011), 1–31. Google ScholarDigital Library
5. Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. 2009. PatchMatch: A Randomized Correspondence Algorithm for Structural Image Editing. ACM Trans. Graph. (Proc. of SIGGRAPH) 28, 3 (2009).Google ScholarDigital Library
6. Connelly Barnes, Eli Shechtman, Dan B Goldman, and Adam Finkelstein. 2010. The Generalized PatchMatch Correspondence Algorithm. In Proc. ECCV.Google ScholarCross Ref
7. Connelly Barnes, Fang-Lue Zhang, Liming Lou, Xian Wu, and Shi-Min Hu. 2015. PatchTable: Efficient Patch Queries for Large Datasets and Applications. ACM Trans. Graph. (Proc. of SIGGRAPH) 34, 4 (2015).Google ScholarDigital Library
8. Pierre Bénard, Forrester Cole, Michael Kass, Igor Mordatch, James Hegarty, Martin Sebastian Senn, Kurt Fleischer, Davide Pesare, and Katherine Breeden. 2013. Stylizing Animation by Example. 32, 4 (2013).Google Scholar
9. T. Brox, C. Bregler, and J. Malik. 2009. Large displacement optical flow. In Proc. CVPR. Google ScholarCross Ref
10. T. Brox, A. Bruhn, N. Papenberg, and J. Weickert. 2004. High accuracy optical flow estimation based on a theory for warping. In Proc. ECCV. Google ScholarCross Ref
11. Dongdong Chen, Lu Yuan, Jing Liao, Nenghai Yu, and Gang Hua. 2017. StyleBank: An Explicit Representation for Neural Image Style Transfer. In Proc. CVPR.Google ScholarCross Ref
12. Tao Chen, Ming-Ming Cheng, Ping Tan, Ariel Shamir, and Shi-Min Hu. 2009. Sketch2Photo: Internet Image Montage. ACM Trans. Graph. (Proc. of SIGGRAPH Asia) 28, 5 (2009).Google Scholar
13. Li Cheng, S.V.N. Vishwanathan, and Xinhua Zhang. 2008. Consistent image analogies using semi-supervised learning. In CVPR. Google ScholarCross Ref
14. Hamilton Y. Chong, Steven J. Gortler, and Todd Zickler. 2008. A Perception-based Color Space for Illumination-invariant Image Processing. ACM Trans. Graph. (Proc. of SIGGRAPH) 27, 3 (2008).Google ScholarDigital Library
15. Cusano Claudio, Gasparini Francesca, and Schettini Raimondo. 2012. Color transfer using semantic image annotation, Vol. 8299.Google Scholar
16. Kevin Dale, Micah K. Johnson, Kalyan Sunkavalli, Wojciech Matusik, and Hanspeter Pfister. 2009. Image restoration using online photo collections. In Proc. ICCV. Google ScholarCross Ref
17. V. Dumoulin, J. Shlens, and M. Kudlur. 2016. A Learned Representation For Artistic Style. arXiv preprint arXiv:1610.07629 (2016).Google Scholar
18. Alexei A Efros and William T Freeman. 2001. Image quilting for texture synthesis and transfer. In Proc. ACM SIGGRAPH. 341–346.Google ScholarDigital Library
19. Zeev Farbman, Raanan Fattal, Dani Lischinski, and Richard Szeliski. 2008. Edge-preserving decompositions for multi-scale tone and detail manipulation. In ACM Trans. Graph. (Proc. of SIGGRAPH), Vol. 27. ACM. Google ScholarDigital Library
20. Oriel Frigo, Neus Sabater, Julie Delon, and Pierre Hellier. 2016. Split and Match: Example-based Adaptive Patch Sampling for Unsupervised Style Transfer. In Proc. CVPR.Google ScholarCross Ref
21. Leon Gatys, Alexander S Ecker, and Matthias Bethge. 2015. Texture synthesis using convolutional neural networks. In Proc. of NIPS.Google Scholar
22. Leon A Gatys, Alexander S Ecker, and Matthias Bethge. 2016. A Neural Algorithm of Artistic Style. In Proc. CVPR.Google ScholarCross Ref
23. Yoav HaCohen, Eli Shechtman, Dan B. Goldman, and Dani Lischinski. 2011. Non-rigid Dense Correspondence with Applications for Image Enhancement. ACM Trans. Graph. (Proc. of SIGGRAPH) 30, 4 (2011).Google ScholarDigital Library
24. Bumsub Ham, Minsu Cho, Cordelia Schmid, and Jean Ponce. 2015. Proposal Flow. arXiv preprint arXiv:1511.05065 (2015).Google Scholar
25. Aaron Hertzmann, Charles E. Jacobs, Nuria Oliver, Brian Curless, and David H. Salesin. 2001. Image Analogies. In Proc. ACM SIGGRAPH. Google ScholarDigital Library
26. Yinlin Hu, Rui Song, and Yunsong Li. 2016. Efficient Coarse-To-Fine PatchMatch for Large Displacement Optical Flow. In Proc. CVPR.Google Scholar
27. Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016. Perceptual losses for real-time style transfer and super-resolution. Proc. ECCV (2016).Google ScholarCross Ref
28. Micah K Johnson, Kevin Dale, Shai Avidan, Hanspeter Pfister, William T Freeman, and Wojciech Matusik. 2011. Cg2real: Improving the realism of computer generated images using a large collection of photographs. IEEE Trans. on Visualization and Computer Graphics 17, 9 (2011).Google ScholarDigital Library
29. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Proc. of NIPS.Google ScholarDigital Library
30. Pierre-Yves Laffont, Zhile Ren, Xiaofeng Tao, Chao Qian, and James Hays. 2014. Transient Attributes for High-level Understanding and Editing of Outdoor Scenes. ACM Trans. Graph. (Proc. of SIGGRAPH) 33, 4 (2014).Google ScholarDigital Library
31. Hochang Lee, Sanghyun Seo, Seungtaek Ryoo, and Kyunghyun Yoon. 2010. Directional Texture Transfer.Google Scholar
32. Chuan Li and Michael Wand. 2016a. Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis. In Proc. CVPR. Google ScholarCross Ref
33. Chuan Li and Michael Wand. 2016b. Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks. arXiv preprint arXiv:1604.04382 (2016).Google Scholar
34. Jing Liao, Rodolfo S. Lima, Diego Nehab, Hugues Hoppe, Pedro V. Sander, and Jinhui Yu. 2014. Automating Image Morphing Using Structural Similarity on a Halfway Domain. ACM Trans. Graph. 33, 5 (2014).Google ScholarDigital Library
35. Ce Liu, Jenny Yuen, and Antonio Torralba. 2011. SIFT Flow: Dense Correspondence Across Scenes and Its Applications. IEEE Trans. Pattern Anal. Mach. Intell. 33, 5 (2011).Google Scholar
36. Jonathan L Long, Ning Zhang, and Trevor Darrell. 2014. Do convnets learn correspondence?. In Proc. of NIPS. 1601–1609.Google Scholar
37. Or Lotan and Michal Irani. 2016. Needle-Match: Reliable Patch Matching Under High Uncertainty. In Proc. CVPR. Google ScholarCross Ref
38. David G. Lowe. 2004. Distinctive Image Features from Scale-Invariant Keypoints. International Journal of Computer Vision 60, 2 (2004). Google ScholarDigital Library
39. Fujun Luan, Sylvain Paris, Eli Shechtman, and Kavita Bala. 2017. Deep Photo Style Transfer. arXiv preprint arXiv:1703.07511 (2017).Google Scholar
40. B. D. Lucas and T. Kanade. 1981. An iterative image registration technique with an application to stereo vision. In Proc. of Imaging Understanding Workshop.Google Scholar
41. Francois Pitie, Anil C. Kokaram, and Rozenn Dahyot. 2005. N-Dimensional Probablility Density Function Transfer and Its Application to Colour Transfer. In Proc. ICCV.Google Scholar
42. Scott E Reed, Yi Zhang, Yuting Zhang, and Honglak Lee. 2015. Deep Visual Analogy-Making. In Proc. of NIPS.Google Scholar
43. Erik Reinhard, Michael Ashikhmin, Bruce Gooch, and Peter Shirley. 2001. Color Transfer Between Images. IEEE Comput. Graph. and Appl. 21, 5 (2001). Google ScholarDigital Library
44. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. 2015. Imagenet large scale visual recognition challenge. International Journal of Computer Vision 115, 3 (2015), 211–252. Google ScholarDigital Library
45. B. C. Russell, J. Sivic, J. Ponce, and H. Dessales. 2011. Automatic Alignment of Paintings and Photographs Depicting a 3D Scene. In 3D Representation for Recognition.Google Scholar
46. Ahmed Selim, Mohamed Elgharib, and Linda Doyle. 2016. Painting Style Transfer for Head Portraits Using Convolutional Neural Networks. ACM Trans. Graph. (Proc. of SIGGRAPH) 35, 4 (2016).Google ScholarDigital Library
47. Eli Shechtman and Michal Irani. 2007. Matching Local Self-Similarities across Images and Videos. In Proc. CVPR. Google ScholarCross Ref
48. Xiaoyong Shen, Xin Tao, Chao Zhou, Hongyun Gao, and Jiaya Jia. 2016. Regional Fore-most Matching for Internet Scene Images. ACM Trans. Graph. (Proc. of SIGGRAPH Asia) 35, 6 (2016).Google Scholar
49. YiChang Shih, Sylvain Paris, Connelly Barnes, William T Freeman, and Frédo Durand. 2014. Style transfer for headshot portraits. ACM Trans. Graph. (Proc. of SIGGRAPH) 33, 4 (2014).Google ScholarDigital Library
50. Yichang Shih, Sylvain Paris, Frédo Durand, and William T Freeman. 2013. Data-driven hallucination of different times of day from a single outdoor photo. ACM Trans. Graph. (Proc. of SIGGRAPH) 32, 6 (2013).Google ScholarDigital Library
51. Abhinav Shrivastava, Tomasz Malisiewicz, Abhinav Gupta, and Alexei A. Efros. 2011. Data-driven Visual Similarity for Cross-domain Image Matching. ACM Trans. Graph. (Proc. of SIGGRAPH) 30, 6 (2011).Google ScholarDigital Library
52. Denis Simakov, Yaron Caspi, Eli Shechtman, and Michal Irani. 2008. Summarizing visual data using bidirectional similarity. In CVPR. Google ScholarCross Ref
53. Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).Google Scholar
54. Yu-Wing Tai, Jiaya Jia, and Chi-Keung Tang. 2005. Local Color Transfer via Probabilistic Segmentation by Expectation-Maximization. In Proc. CVPR.Google Scholar
55. Dmitry Ulyanov, Vadim Lebedev, Andrea Vedaldi, and Victor Lempitsky. 2016. Texture Networks: Feed-forward Synthesis of Textures and Stylized Images. arXiv preprint arXiv:1603.03417 (2016).Google Scholar
56. Philippe Weinzaepfel, Jérôme Revaud, Zaid Harchaoui, and Cordelia Schmid. 2013. DeepFlow: Large displacement optical flow with deep matching. In Proc. ICCV. Google ScholarDigital Library
57. Tomihisa Welsh, Michael Ashikhmin, and Klaus Mueller. 2002. Transferring Color to Greyscale Images. Proc. ACM SIGGRAPH 21, 3 (2002).Google ScholarDigital Library
58. Fuzhang Wu, Weiming Dong, Yan Kong, Xing Mei, Jean-Claude Paul, and Xiaopeng Zhang. 2013. Content-Based Colour Transfer. Computer Graphics Forum (2013).Google Scholar
59. Yu Xiang, Roozbeh Mottaghi, and Silvio Savarese. 2014. Beyond pascal: A benchmark for 3d object detection in the wild. In Applications of Computer Vision (WACV), 2014 IEEE Winter Conference on. IEEE, 75–82.Google ScholarCross Ref
60. Gehua Yang, Charles V Stewart, Michal Sofka, and Chia-Ling Tsai. 2007. Registration of challenging image pairs: Initialization, estimation, and decision. IEEE transactions on pattern analysis and machine intelligence 29, 11 (2007).Google Scholar
61. Hongsheng Yang, Wen-Yan Lin, and Jiangbo Lu. 2014. DAISY Filter Flow: A Generalized Discrete Approach to Dense Correspondences. In Proc. CVPR. Google ScholarDigital Library
62. Matthew D. Zeiler and Rob Fergus. 2014. Visualizing and understanding convolutional networks. In Proc. ECCV. Google ScholarCross Ref
63. Wei Zhang, Chen Cao, Shifeng Chen, Jianzhuang Liu, and Xiaoou Tang. 2013. Style Transfer Via Image Component Analysis. IEEE Trans. on Multimedia (2013).Google Scholar
64. Tinghui Zhou, Philipp Krähenbühl, Mathieu Aubry, Qixing Huang, and Alexei A. Efros. 2016. Learning Dense Correspondence via 3D-guided Cycle Consistency. In Proc. CVPR. Google ScholarCross Ref
65. Ciyou Zhu, Richard H. Byrd, Peihuang Lu, and Jorge Nocedal. 1994. L-BFGS-B: Fortran Subroutines for Large-Scale Bound Constrained Optimization. (1994).Google Scholar
