“Data-driven hallucination of different times of day from a single outdoor photo” by Shih, Paris, Durand and Freeman
Conference:
Type(s):
Title:
- Data-driven hallucination of different times of day from a single outdoor photo
Session/Category Title: HDR & IBR
Presenter(s)/Author(s):
Abstract:
We introduce “time hallucination”: synthesizing a plausible image at a different time of day from an input image. This challenging task often requires dramatically altering the color appearance of the picture. In this paper, we introduce the first data-driven approach to automatically creating a plausible-looking photo that appears as though it were taken at a different time of day. The time of day is specified by a semantic time label, such as “night”.Our approach relies on a database of time-lapse videos of various scenes. These videos provide rich information about the variations in color appearance of a scene throughout the day. Our method transfers the color appearance from videos with a similar scene as the input photo. We propose a locally affine model learned from the video for the transfer, allowing our model to synthesize new color data while retaining image details. We show that this model can hallucinate a wide range of different times of day. The model generates a large sparse linear system, which can be solved by off-the-shelf solvers. We validate our methods by synthesizing transforming photos of various outdoor scenes to four times of interest: daytime, the golden hour, the blue hour, and nighttime.
References:
1. Barnes, C., Shechtman, E., Goldman, D., and Finkelstein, A. 2010. The generalized patchmatch correspondence algorithm. European Conference on Computer Vision, 29–43.
2. Bousseau, A., Paris, S., and Durand, F. 2009. User-assisted intrinsic images. In ACM Transactions on Graphics (SIGGRAPH), vol. 28, 130.
3. Bychkovsky, V., Paris, S., Chan, E., and Durand, F. 2011. Learning photographic global tonal adjustment with a database of input/output image pairs. In IEEE Conference on Computer Vision and Pattern Recognition, 97–104.
4. Caputo, R. 2005. In Potography field guide, National Geographics, 104–115.
5. Dalal, N., and Triggs, B. 2005. Histograms of oriented gradients for human detection. In IEEE Conference on Computer Vision and Pattern Recognition, vol. 1, 886–893.
6. Efros, A. A., and Freeman, W. T. 2001. Image quilting for texture synthesis and transfer. In Annual Conference on Computer Graphics and Interactive Techniques, 341–346.
7. Eisemann, E., and Durand, F. 2004. Flash photography enhancement via intrinsic relighting. In ACM Transactions on Graphics (SIGGRAPH), vol. 23, 673–678.
8. Freeman, W., Pasztor, E., and Carmichael, O. 2000. Learning low-level vision. International Journal of Computer Vision 40, 1, 25–47.
9. Freeman, W., Jones, T., and Pasztor, E. 2002. Example-based super-resolution. IEEE Computer Graphics and Applications 22, 2, 56–65.
10. HaCohen, Y., Shechtman, E., Goldman, D., and Lischinski, D. 2011. Non-rigid dense correspondence with applications for image enhancement. ACM Transactions on Graphics (SIGGRAPH) 30, 4, 70.
11. Hays, J., and Efros, A. 2007. Scene completion using millions of photographs. In ACM Transactions on Graphics (SIGGRAPH), vol. 26, 4.
12. He, K., Sun, J., and Tang, X. 2010. Guided image filtering. European Conference on Computer Vision, 1–14.
13. Hertzmann, A., Jacobs, C., Oliver, N., Curless, B., and Salesin, D. 2001. Image analogies. In Conference on Computer Graphics and Interactive Techniques, 327–340.
14. Irony, R., Cohen-Or, D., and Lischinski, D. 2005. Colorization by example. In Eurographics Conference on Rendering Techniques, 201–210.
15. Jacobs, N., Roman, N., and Pless, R. 2007. Consistent temporal variations in many outdoor scenes. In IEEE Conference on Computer Vision and Pattern Recognition, 1–6.
16. Kopf, J., Neubert, B., Chen, B., Cohen, M., Cohen-Or, D., Deussen, O., Uyttendaele, M., and Lischinski, D. 2008. Deep photo: Model-based photograph enhancement and viewing. In ACM Transactions on Graphics (SIGGRAPH), vol. 27, 116.
17. Laffont, P.-Y., Bousseau, A., Paris, S., Durand, F., Drettakis, G., et al. 2012. Coherent intrinsic images from photo collections. ACM Transactions on Graphics 31, 6.
18. Lalonde, J., Efros, A., and Narasimhan, S. 2009. Webcam clip art: Appearance and illuminant transfer from time-lapse sequences. In ACM Transactions on Graphics (SIGGRAPH), vol. 28, 131.
19. Levin, A., Lischinski, D., and Weiss, Y. 2004. Colorization using optimization. In ACM Transactions on Graphics (SIGGRAPH), vol. 23, 689–694.
20. Levin, A., Lischinski, D., and Weiss, Y. 2006. A closed form solution to natural image matting. In IEEE Conference on Computer Vision and Pattern Recognition, vol. 1, 61–68.
21. Liu, C., Yuen, J., Torralba, A., Sivic, J., and Freeman, W. 2008. Sift flow: Dense correspondence across different scenes. European Conference on Computer Vision, 28–42.
22. Petschnigg, G., Szeliski, R., Agrawala, M., Cohen, M., Hoppe, H., and Toyama, K. 2004. Digital photography with flash and no-flash image pairs. In ACM Transactions on Graphics (SIGGRAPH), vol. 23, 664–672.
23. Pitie, F., Kokaram, A., and Dahyot, R. 2005. N-dimensional probability density function transfer and its application to color transfer. In IEEE International Conference on Computer Vision, vol. 2, 1434–1439.
24. Pouli, T., and Reinhard, E. 2011. Progressive color transfer for images of arbitrary dynamic range. Computers and Graphics 35, 1, 67–80.
25. Reinhard, E., Ashikhmin, M., Gooch, B., and Shirley, P. 2001. Color transfer between images. IEEE Computer Graphics and Applications 21, 5, 34–41.
26. Rowell, G. 2012. In Mountain Light, Sierra Club Books. Xiao, J., Hays, J., Ehinger, K., Oliva, A., and Torralba, A. 2010. Sun database: Large-scale scene recognition from abbey to zoo. In IEEE conference on Computer Vision and Pattern Recognition, 3485–3492.
27. Yedidia, J. S., Freeman, W. T., Weiss, Y., et al. 2000. Generalized belief propagation. In Advances in Neural Information Processing Systems, vol. 13, 689–695.
