“Time-travel rephotography” by Luo, Zhang and Yoo
Conference:
Type(s):
Title:
- Time-travel rephotography
Session/Category Title: Computational Photography
Presenter(s)/Author(s):
Abstract:
Many historical people were only ever captured by old, faded, black and white photos, that are distorted due to the limitations of early cameras and the passage of time. This paper simulates traveling back in time with a modern camera to rephotograph famous subjects. Unlike conventional image restoration filters which apply independent operations like denoising, colorization, and superresolution, we leverage the StyleGAN2 framework to project old photos into the space of modern high-resolution photos, achieving all of these effects in a unified framework. A unique challenge with this approach is retaining the identity and pose of the subject in the original photo, while discarding the many artifacts frequently seen in low-quality antique photos. Our comparisons to current state-of-the-art restoration filters show significant improvements and compelling results for a variety of important historical people. Please go to time-travell-rephotography.github.io for many more results.
References:
1. Rameen Abdal, Yipeng Qin, and Peter Wonka. 2019. Image2stylegan: How to embed images into the stylegan latent space?. In Proceedings of the IEEE International Conference on Computer Vision. 4432–4441.
2. Rameen Abdal, Yipeng Qin, and Peter Wonka. 2020. Image2StyleGAN++: How to Edit the Embedded Images?. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8296–8305.
3. Ansel Adams. 2018. The negative. Ansel Adams. 21–25 pages.
4. Jason Antic. 2019. jantic/deoldify: A deep learning based project for colorizing and restoring old images (and video!). https://github.com/jantic/DeOldify
5. S Derin Babacan, Rafael Molina, and Aggelos K Katsaggelos. 2008. Total variation super resolution using a variational approach. In IEEE International Conference on Image Processing. IEEE, 641–644.
6. Peter Baylies. 2019. Stylegan encoder – converts real images to latent space. https://github.com/pbaylies/stylegan-encoder
7. Nick Brandreth. 2021. Nick Brandreth. https://www.nickbrandreth.com/.
8. Antoni Buades, Bartomeu Coll, and J-M Morel. 2005. A non-local algorithm for image denoising. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Vol. 2. IEEE, 60–65.
9. Adrian Bulat and Georgios Tzimiropoulos. 2018. Super-fan: Integrated facial landmark localization and super-resolution of real-world low resolution faces in arbitrary poses with gans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 109–117.
10. Wilhelm Burger and Mark J Burge. 2016. Digital image processing: an algorithmic introduction using Java. Springer.
11. Guillaume Charpiat, Matthias Hofmann, and Bernhard Schölkopf. 2008. Automatic image colorization via multimodal predictions. In Proceedings of the European Conference on Computer Vision. Springer, 126–139.
12. Zezhou Cheng, Qingxiong Yang, and Bin Sheng. 2015. Deep colorization. In Proceedings of the IEEE International Conference on Computer Vision. 415–423.
13. Alex Yong-Sang Chia, Shaojie Zhuo, Raj Kumar Gupta, Yu-Wing Tai, Siu-Yeung Cho, Ping Tan, and Stephen Lin. 2011. Semantic colorization with internet images. ACM Transactions on Graphics (TOG) 30, 6 (2011), 1–8.
14. Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. 2007. Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Transactions on image Processing 16, 8 (2007), 2080–2095.
15. Aditya Deshpande, Jason Rock, and David Forsyth. 2015. Learning large-scale automatic image colorization. In Proceedings of the IEEE International Conference on Computer Vision. 567–575.
16. Chao Dong, Yubin Deng, Chen Change Loy, and Xiaoou Tang. 2015a. Compression artifacts reduction by a deep convolutional network. In Proceedings of the IEEE International Conference on Computer Vision. 576–584.
17. Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. 2015b. Image super-resolution using deep convolutional networks. IEEE transactions on pattern analysis and machine intelligence 38, 2 (2015), 295–307.
18. Michael Elad and Michal Aharon. 2006. Image denoising via sparse and redundant representations over learned dictionaries. IEEE Transactions on Image Processing 15, 12 (2006), 3736–3745.
19. Leon A Gatys, Alexander S Ecker, and Matthias Bethge. 2016. Image style transfer using convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2414–2423.
20. Joe Geigel and F Kenton Musgrave. 1997. A model for simulating the photographic development process on digital images. In Proceedings of the annual Conference on Computer graphics and interactive techniques. 135–142.
21. Klemen Grm, Walter J Scheirer, and Vitomir Štruc. 2019. Face hallucination using cascaded super-resolution and identity priors. IEEE Transactions on Image Processing 29, 1 (2019), 2150–2165.
22. Jun Guo and Hongyang Chao. 2016. Building dual-domain representations for compression artifacts reduction. In Proceedings of the European Conference on Computer Vision. Springer, 628–644.
23. Raj Kumar Gupta, Alex Yong-Sang Chia, Deepu Rajan, Ee Sin Ng, and Huang Zhiyong. 2012. Image colorization using similar images. In Proceedings of the ACM International Conference on Multimedia. 369–378.
24. Yoav Hacohen, Eli Shechtman, and Dani Lischinski. 2013. Deblurring by example using dense correspondence. In Proceedings of the IEEE International Conference on Computer Vision. 2384–2391.
25. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 770–778.
26. Mingming He, Dongdong Chen, Jing Liao, Pedro V Sander, and Lu Yuan. 2018. Deep exemplar-based colorization. ACM Transactions on Graphics (TOG) 37, 4 (2018), 1–16.
27. Yi-Chin Huang, Yi-Shin Tung, Jun-Cheng Chen, Sung-Wen Wang, and Ja-Ling Wu. 2005. An adaptive edge detection based colorization algorithm and its applications. In Proceedings of the annual ACM International Conference on Multimedia. 351–354.
28. Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. 2016. Let there be color! Joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification. ACM Transactions on Graphics (ToG) 35, 4 (2016), 1–11.
29. Revital Ironi, Daniel Cohen-Or, and Dani Lischinski. 2005. Colorization by Example.. In Rendering Techniques. Citeseer, 201–210.
30. Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1125–1134.
31. Henrik Wann Jensen, Stephen R Marschner, Marc Levoy, and Pat Hanrahan. 2001. A practical model for subsurface light transport. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques. 511–518.
32. Kimmo Karkkainen and Jungseock Joo. 2021. FairFace: Face Attribute Dataset for Balanced Race, Gender, and Age for Bias Measurement and Mitigation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 1548–1558.
33. Tero Karras, Samuli Laine, and Timo Aila. 2019. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4401–4410.
34. Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. 2020a. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8110–8119.
35. Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. 2020b. Analyzing and Improving the Image Quality of StyleGAN. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
36. Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. 2016. Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1646–1654.
37. Davis E. King. 2009. Dlib-ml: A Machine Learning Toolkit. Journal of Machine Learning Research 10 (2009), 1755–1758.
38. Orest Kupyn, Volodymyr Budzan, Mykola Mykhailych, Dmytro Mishkin, and Jiří Matas. 2018. Deblurgan: Blind motion deblurring using conditional adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8183–8192.
39. Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. 2016. Learning representations for automatic colorization. In Proceedings of the European Conference on Computer Vision. Springer, 577–593.
40. Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. 2017. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4681–4690.
41. Stamatios Lefkimmiatis. 2017. Non-local color image denoising with convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3587–3596.
42. Anat Levin, Dani Lischinski, and Yair Weiss. 2004. Colorization using optimization. In ACM SIGGRAPH 2004 Papers. 689–694.
43. Zhen Li, Jinglei Yang, Zheng Liu, Xiaomin Yang, Gwanggil Jeon, and Wei Wu. 2019. Feedback network for image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3867–3876.
44. Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. 2020. On the Variance of the Adaptive Learning Rate and Beyond. In Proceedings of the Eighth International Conference on Learning Representations (ICLR 2020).
45. Xiaopei Liu, Liang Wan, Yingge Qu, Tien-Tsin Wong, Stephen Lin, Chi-Sing Leung, and Pheng-Ann Heng. 2008. Intrinsic colorization. In ACM SIGGRAPH Asia 2008 papers. 1–9.
46. Qing Luan, Fang Wen, Daniel Cohen-Or, Lin Liang, Ying-Qing Xu, and Heung-Yeung Shum. 2007. Natural image colorization. In Proceedings of the Eurographics Conference on Rendering Techniques. 309–320.
47. Roey Mechrez, Itamar Talmi, and Lihi Zelnik-Manor. 2018. The contextual loss for image transformation with non-aligned data. In Proceedings of the European Conference on Computer Vision. 768–783.
48. MediaWiki. 2021. MediaWiki: PageImages. https://www.mediawiki.org/wiki/Extension:PageImages.
49. Sachit Menon, Alex Damian, McCourt Hu, Nikhil Ravi, and Cynthia Rudin. 2020a. PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
50. Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi, and Cynthia Rudin. 2020b. PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2437–2445.
51. Anish Mittal, Rajiv Soundararajan, and Alan C Bovik. 2012. Making a “completely blind” image quality analyzer. IEEE Signal processing letters 20, 3 (2012), 209–212.
52. Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. 2017. Deep multi-scale convolutional neural network for dynamic scene deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3883–3891.
53. Beaumont Newhall. 1982. The History of Photography: From 1839 to the Present (5 ed.). The Museum of Modern Art.
54. State Library of South Australia. 2021. Photo of John McDouall Stuart. https://www.catalog.slsa.sa.gov.au/record=b2049594~S1.
55. Jinshan Pan, Zhe Hu, Zhixun Su, and Ming-Hsuan Yang. 2014. Deblurring face images with exemplars. In Proceedings of the European Conference on Computer Vision. Springer, 47–62.
56. Xingang Pan, Xiaohang Zhan, Bo Dai, Dahua Lin, Chen Change Loy, and Ping Luo. 2020. Exploiting Deep Generative Prior for Versatile Image Restoration and Manipulation. In European Conference on Computer Vision (ECCV).
57. Omkar M. Parkhi, Andrea Vedaldi, and Andrew Zisserman. 2015. Deep Face Recognition. In British Machine Vision Conference.
58. Yingge Qu, Tien-Tsin Wong, and Pheng-Ann Heng. 2006. Manga colorization. ACM Transactions on Graphics (TOG) 25, 3 (2006), 1214–1220.
59. Wenqi Ren, Jiaolong Yang, Senyou Deng, David Wipf, Xiaochun Cao, and Xin Tong. 2019. Face video deblurring using 3D facial priors. In Proceedings of the IEEE International Conference on Computer Vision. 9388–9397.
60. Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or. 2020. Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation. arXiv preprint arXiv:2008.00951 (2020).
61. Joni Salminen, Soon-gyo Jung, Shammur Chowdhury, and Bernard J Jansen. 2020. Analyzing demographic bias in artificially generated facial pictures. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems. 1–8.
62. Ziyi Shen, Wei-Sheng Lai, Tingfa Xu, Jan Kautz, and Ming-Hsuan Yang. 2018. Deep semantic face deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8260–8269.
63. Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).
64. Jheng-Wei Su, Hung-Kuo Chu, and Jia-Bin Huang. 2020. Instance-aware Image Colorization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
65. Masanori Suganuma, Xing Liu, and Takayuki Okatani. 2019. Attention-based adaptive selection of operations for image restoration in the presence of unknown combined distortions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 9039–9048.
66. Jian Sun, Wenfei Cao, Zongben Xu, and Jean Ponce. 2015. Learning a convolutional neural network for non-uniform motion blur removal. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 769–777.
67. Daniel Sỳkora, John Dingliana, and Steven Collins. 2009. Lazybrush: Flexible painting tool for hand-drawn cartoons. Computer Graphics Forum 28, 2 (2009), 599–608.
68. Ying Tai, Jian Yang, Xiaoming Liu, and Chunyan Xu. 2017. Memnet: A persistent memory network for image restoration. In Proceedings of the IEEE International Conference on Computer Vision. 4539–4547.
69. Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or. 2021. Designing an Encoder for StyleGAN Image Manipulation. arXiv preprint arXiv:2102.02766 (2021).
70. Ziyu Wan, Bo Zhang, Dongdong Chen, Pan Zhang, Dong Chen, Jing Liao, and Fang Wen. 2020. Old Photo Restoration via Deep Latent Space Translation. arXiv:arXiv:2009.07047
71. Xintao Wang, Yu Li, Honglun Zhang, and Ying Shan. 2021. Towards Real-World Blind Face Restoration with Generative Facial Prior. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
72. Zhangyang Wang, Ding Liu, Shiyu Chang, Qing Ling, Yingzhen Yang, and Thomas S Huang. 2016. D3: Deep dual-domain based fast restoration of JPEG-compressed images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2764–2772.
73. Tomihisa Welsh, Michael Ashikhmin, and Klaus Mueller. 2002. Transferring color to greyscale images. In Proceedings of the Conference on Computer graphics and interactive techniques. 277–280.
74. Wikipedia. 2021a. Dry Plate. https://en.wikipedia.org/wiki/Dry_plate.
75. Wikipedia. 2021b. Wikipedia: Photographic film. https://en.wikipedia.org/wiki/Photographic_film.
76. Wikipedia. 2021c. Wikipedia: Rephotography. https://en.wikipedia.org/wiki/Rephotography.
77. Junyuan Xie, Linli Xu, and Enhong Chen. 2012. Image denoising and inpainting with deep neural networks. In Advances in neural information Processing systems. 341–349.
78. Li Xu, Jimmy SJ Ren, Ce Liu, and Jiaya Jia. 2014a. Deep convolutional neural network for image deconvolution. In Advances in Neural Information Processing Systems. 1790–1798.
79. Li Xu, Xin Tao, and Jiaya Jia. 2014b. Inverse kernels for fast spatial deconvolution. In Proceedings of the European Conference on Computer Vision. Springer, 33–48.
80. Jianchao Yang, John Wright, Thomas S Huang, and Yi Ma. 2010. Image super-resolution via sparse representation. IEEE transactions on image Processing 19, 11 (2010), 2861–2873.
81. Tao Yang, Ren Peiran, Xie Xuansong, and Lei Zhang. 2021. GAN Prior Embedded Network for Blind Face Restoration in the Wild. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
82. Liron Yatziv and Guillermo Sapiro. 2006. Fast image and video colorization using chrominance blending. IEEE transactions on Image Processing 15, 5 (2006), 1120–1129.
83. Ke Yu, Chao Dong, Liang Lin, and Chen Change Loy. 2018. Crafting a Toolchain for Image Restoration by Deep Reinforcement Learning. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2443–2452.
84. Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. 2017b. Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE Transactions on Image Processing 26, 7 (2017), 3142–3155.
85. Kai Zhang, Wangmeng Zuo, Shuhang Gu, and Lei Zhang. 2017c. Learning deep CNN denoiser prior for image restoration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3929–3938.
86. Kai Zhang, Wangmeng Zuo, and Lei Zhang. 2018b. FFDNet: Toward a fast and flexible solution for CNN-based image denoising. IEEE Transactions on Image Processing 27, 9 (2018), 4608–4622.
87. Richard Zhang, Phillip Isola, and Alexei A Efros. 2016. Colorful image colorization. In Proceedings of the European Conference on Computer Vision. Springer, 649–666.
88. Richard Zhang, Jun-Yan Zhu, Phillip Isola, Xinyang Geng, Angela S Lin, Tianhe Yu, and Alexei A Efros. 2017a. Real-time user-guided image colorization with learned deep priors. ACM Transactions on Graphics (TOG) 36, 4 (2017), 1–11.
89. Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. 2018a. Residual dense network for image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2472–2481.
90. Jiaojiao Zhao, Li Liu, Cees GM Snoek, Jungong Han, and Ling Shao. 2018. Pixel-level semantics guided image colorization. arXiv preprint arXiv:1808.01597 (2018).
91. Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou. 2020. In-domain GAN Inversion for Real Image Editing. In Proceedings of European Conference on Computer Vision (ECCV).
92. Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A Efros. 2016. Generative visual manipulation on the natural image manifold. In European conference on computer vision. Springer, 597–613.
