“Interactive Relighting in Single Low-Dynamic Range Images” by Wu and Saito
Conference:
Type(s):
Title:
- Interactive Relighting in Single Low-Dynamic Range Images
Session/Category Title: Image and Light Field Manipulation
Presenter(s)/Author(s):
Moderator(s):
Abstract:
This article addresses the relighting of outdoor and large indoor scenes illuminated by nondistant lights, which has seldom been discussed in previous works. We propose a method for users to interactively edit the illumination of a scene by moving existing lights and inserting synthetic lights into the scene that requires only a small amount of user annotation and a single low-dynamic range (LDR) image. We achieve this by adopting a top-down approach that estimates the scene reflectance by fitting a diffuse illumination model to a photograph. This approach gains stability and robustness by estimating the camera, scene geometry, and light sources in sequence and by using a confidence map, which is a per-pixel weight map. The results of our evaluation demonstrates that the proposed method can estimate a scene accurately enough for realistic relighting of images. Moreover, the experimental results of our user studies show that the synthesized images are so realistic as to be almost indistinguishable from real photographs.
References:
1. Cureos Auders Gustafsson. 2012. Constrained optimization by linear approximation for java. (2012). http://www.codeproject.com/Articles/508513/Derivative-free-nonlinear-optimization-for-NET-and-optimization-for-NET-and.Google Scholar
2. Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. 2009. PatchMatch: A randomized correspondence algorithm for structural image editing. ACM Trans. Graph. 28, 3, Article 24 (July 2009), 11 pages. Google ScholarDigital Library
3. Jonathan T. Barron and Jitendra Malik. 2012. Color constancy, intrinsic images, and shape estimation. In ECCV’12. 57–70. Google ScholarDigital Library
4. Sean Bell, Kavita Bala, and Noah Snavely. 2014. Intrinsic images in the wild. ACM Trans. Graph. 33, 4, Article 159 (July 2014), 12 pages. Google ScholarDigital Library
5. Blender. 2015. 2.72 Release. https://www.blender.org/. (2015).Google Scholar
6. Samuel Boivin and Andre Gagalowicz. 2001. Image-based rendering of diffuse, specular and glossy surfaces from a single image. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques. 107–116. Google ScholarDigital Library
7. Adrien Bousseau, Sylvain Paris, and Frédo Durand. 2009. User-assisted intrinsic images. ACM Trans. Graph. 28, 5 (Dec. 2009), 130:1–130:10. Google ScholarDigital Library
8. Ivaylo Boyadzhiev, Sylvain Paris, and Kavita Bala. 2013. User-assisted image compositing for photographic lighting. ACM Trans. Graph. 32, 4, Article 36 (July 2013), 12 pages. Google ScholarDigital Library
9. Xiaochun Cao and Mubarak Shah. 2005. Camera calibration and light source estimation from images with shadows. In IEEE CVPR’05. 918–923. Google ScholarDigital Library
10. Robert Carroll, Ravi Ramamoorthi, and Maneesh Agrawala. 2011. Illumination decomposition for material recoloring with consistent interreflections. ACM Trans. Graph. 30, 4 (July 2011), 43:1–43:10. Google ScholarDigital Library
11. Manmohan Chandraker and Ravi Ramamoorthi. 2011. What an image reveals about material reflectance. In IEEE ICCV’11. 1076–1083. Google ScholarDigital Library
12. Alex Colburn, Aseem Agarwala, Aaron Hertzmann, Brian Curless, and Michael F. Cohen. 2013. Image-based remodeling. IEEE TVCG 19, 1 (2013), 56–66. Google ScholarDigital Library
13. A. Criminisi, I. Reid, and A. Zisserman. 2000. Single view metrology. Int. J. Comput. Vision 40, 2 (Nov. 2000), 123–148. Google ScholarDigital Library
14. Yue Dong, Xin Tong, Fabio Pellacini, and Baining Guo. 2011. AppGen: Interactive material modeling from a single image. ACM Trans. Graph. 30, 6, Article 146 (Dec. 2011), 10 pages. Google ScholarDigital Library
15. Sabry El-hakim, Emily Whiting, and Lorenzo Gonzo. 2005. 3D modelling with reusable and integrated building blocks. In 7th Conference on Optical 3-D Measurement Techniques. 3–5.Google Scholar
16. Elena Garces, Adolfo Munoz, Jorge Lopez-Moreno, and Diego Gutierrez. 2012. Intrinsic images by clustering. Comput. Graph. Forum 31, 4 (June 2012), 1415–1424. Google ScholarDigital Library
17. Roger Grosse, Micah K. Johnson, Edward H. Adelson, and William T. Freeman. 2009. Ground-truth dataset and baseline evaluations for intrinsic image algorithms. In IEEE ICCV’09. 2335–2342.Google Scholar
18. Mohammed Hachama, Bernard Ghanem, and Peter Wonka. 2015. Intrinsic scene decomposition from RGB-D images. In IEEE ICCV’15. 810–818. Google ScholarDigital Library
19. Kenji Hara, Ko Nishino, and Katsushi Ikeuchi. 2005. Light source position and reflectance estimation from a single view without the distant illumination assumption. IEEE TPAMI 27, 4 (April 2005), 493–505. Google ScholarDigital Library
20. Kenji Hara, Ko Nishino, and Katsushi Ikeuchi. 2008. Mixture of spherical distributions for single-view relighting. IEEE TPAMI 30, 1 (Jan. 2008), 25–35. Google ScholarDigital Library
21. Richard Hartley and Andrew Zisserman. 2004. Multiple View Geometry in Computer Vision (2nd ed.). Cambridge University Press. Google ScholarDigital Library
22. Berthold K. P. Horn. 1989. Shape from shading. MIT Press, Cambridge, MA, Chapter Obtaining Shape from Shading Information, 123–171. http://dl.acm.org/citation.cfm?id=93871.93877 Google ScholarDigital Library
23. Eugene Hsu, Tom Mertens, Sylvain Paris, Shai Avidan, and Frédo Durand. 2008. Light mixture estimation for spatially varying white balance. ACM Trans. Graph. 27, 3, Article 70 (Aug. 2008), 7 pages. Google ScholarDigital Library
24. Satoshi Iizuka, Yoshihiro Kanamori, Jun Mitani, and Yukio Fukui. 2011. Efficiently modeling 3d scenes from a single image. IEEE CG&A 32, 6 (2011), 18–25. Google ScholarDigital Library
25. Ennio Mingolla James T. Todd. 1983. Perception of surface curvature and direction of illumination from patterns of shading. Journal of Experimental Psychology: Human Perception and Performance 9, 4 (1983), 583–595.Google ScholarCross Ref
26. Andrea J. van Doorm Jan J. Koenderink and Sylvia C. Pont. 2004. Light direction from shad(ow)ed random gaussian surfaces. Perception 33, 12 (2004), 1405–1420.Google ScholarCross Ref
27. Nianjuan Jiang, Ping Tan, and Loong-Fah Cheong. 2009. Symmetric architecture modeling with a single image. ACM Trans. Graph. 28, 5, Article 113 (Dec. 2009), 8 pages. Google ScholarDigital Library
28. Micah K. Johnson and Hany Farid. 2005. Exposing digital forgeries by detecting inconsistencies in lighting. In Proceedings of the 7th Workshop on Multimedia and Security. 1–10. Google ScholarDigital Library
29. Kevin Karsch, Varsha Hedau, David Forsyth, and Derek Hoiem. 2011. Rendering synthetic objects into legacy photographs. ACM Trans. Graph. 30, 6, Article 157 (Dec. 2011), 12 pages. Google ScholarDigital Library
30. Eric Kee, James F. O’Brien, and Hany Farid. 2013. Exposing photo manipulation with inconsistent shadows. ACM Trans. Graph. 32, 3 (July 2013), 28:1–28:12. Google ScholarDigital Library
31. Joel Kronander, Francesco Banterle, Andrew Gardner, Ehsan Miandji, and Jonas Unger. 2015. Photorealistic rendering of mixed reality scenes. Comput. Graph. Forum 34, 2 (May 2015), 643–665. Google ScholarDigital Library
32. Pascal Lagger and Pascal Fua. 2006. Using specularities to recover multiple light sources in the presence of texture. In ICPR. 587–590. http://dblp.uni-trier.de/db/conf/icpr/icpr2006-1.html#LaggerF06 Google ScholarDigital Library
33. Pascal Lagger and Pascal Fua. 2008. Retrieving multiple light sources in the presence of specular reflections and texture. Comput. Vis. Image Underst. 111, 2 (Aug. 2008), 207–218. Google ScholarDigital Library
34. Jean-François Lalonde, Alexei A. Efros, and Srinivasa G. Narasimhan. 2012. Estimating the natural illumination conditions from a single outdoor image. Int. J. Comput. Vision 98, 2 (June 2012), 123–145. Google ScholarDigital Library
35. Edwin H. Land, John, and J. Mccann. 1971. Lightness and retinex theory. Journal of the Optical Society of America 61, 1 (1971), 1–11.Google ScholarCross Ref
36. Hao Li, Etienne Vouga, Anton Gudym, Linjie Luo, Jonathan T. Barron, and Gleb Gusev. 2013. 3D self-portraits. ACM Trans. Graph. 32, 6 (Nov. 2013), 187:1–187:9. Google ScholarDigital Library
37. Yanli Liu, Xueying Qin, Songhua Xu, Eihachiro Nakamae, and Qunsheng Peng. 2009. Light source estimation of outdoor scenes for mixed reality. Vis. Comput. 25, 5 (April 2009), 637–646. Google ScholarDigital Library
38. Stephen Lombardi and Ko Nishino. 2012. Reflectance and natural illumination from a single image. In ECCV’12. 582–595. Google ScholarDigital Library
39. Jorge Lopez-Moreno, Elena Garces, Sunil Hadap, Erik Reinhard, and Diego Gutierrez. 2013. Multiple light source estimation in a single image. Comput. Graph. Forum 32, 8 (2013), 170–182.Google ScholarCross Ref
40. Jorge Lopez-Moreno, Sunil Hadap, Erik Reinhard, and Diego Gutierrez. 2010a. Compositing images through light source detection. Computers & Graphics 34, 6 (2010), 698–707. Google ScholarDigital Library
41. Jorge Lopez-Moreno, Veronica Sundstedt, Francisco Sangorrin, and Diego Gutierrez. 2010b. Measuring the perception of light inconsistencies. In ACM APGV’10. 25–32. Google ScholarDigital Library
42. Wojciech Matusik, Hanspeter Pfister, Matt Brand, and Leonard McMillan. 2003. A data-driven reflectance model. ACM Trans. Graph. 22, 3 (July 2003), 759–769. Google ScholarDigital Library
43. Byong Mok Oh, Max Chen, Julie Dorsey, and Frédo Durand. 2001. Image-based modeling and photo editing. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques. 433–442. Google ScholarDigital Library
44. Alexandros Panagopoulos, Dimitris Samaras, and Nikos Paragios. 2009. Robust shadow and illumination estimation using a mixture model. In IEEE CVPR’09. 651–658.Google Scholar
45. Michael J. D. Powell. 1964. An efficient method for finding the minimum of a function of several variables without calculating derivatives. Computer Journal 7, 2 (1964), 155–162.Google ScholarCross Ref
46. Peiran Ren, Yue Dong, Stephen Lin, Xin Tong, and Baining Guo. 2015. Image based relighting using neural networks. ACM Trans. Graph. 34, 4, Article 111 (July 2015), 12 pages. Google ScholarDigital Library
47. Li Shen, Ping Tan, and Stephen Lin. 2008. Intrinsic image decomposition with non-local texture cues. In IEEE CVPR’08. 1–7. http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=4587660Google Scholar
48. Jian Shi, Yue Dong, Xin Tong, and Yanyun Chen. 2015. Efficient intrinsic image decomposition for RGBD images. In ACM VRST’15. 17–25. Google ScholarDigital Library
49. Borom Tunwattanapong, Abhijeet Ghosh, and Paul Debevec. 2011. Practical image-based relighting and editing with spherical-harmonics and local lights. In CVMP’11. 138–147. Google ScholarDigital Library
50. Lvdi Wang, Li-Yi Wei, Kun Zhou, Baining Guo, and Heung-Yeung Shum. 2007. High dynamic range image hallucination. In Eurographics Symposium on Rendering’07. 321–326. Google ScholarDigital Library
51. Tai-Pang Wu, Chi-Keung Tang, Michael S. Brown, and Heung-Yeung Shum. 2007. Natural shadow matting. ACM Trans. Graph. 26, 2, Article 8 (June 2007). Google ScholarDigital Library
52. S. Xu and A. M. Wallace. 2008. Recovering surface reflectance and multiple light locations and intensities from image data. Pattern Recogn. Lett. 29, 11 (Aug. 2008), 1639–1647. Google ScholarDigital Library
53. Yizhou Yu, Paul Debevec, Jitendra Malik, and Tim Hawkins. 1999. Inverse global illumination: Recovering reflectance models of real scenes from photographs. In ACM SIGGRAPH’99. 215–224. Google ScholarDigital Library
54. Patrick Cavanagh Yuri Ostrovsky and Pawan Sinha. 2005. Perceiving illumination inconsistencies in scenes. Perception 34, 11 (Nov. 2005), 1301–1314.Google Scholar
55. Youyi Zheng, Xiang Chen, Ming-Ming Cheng, Kun Zhou, Shi-Min Hu, and Niloy J. Mitra. 2012. Interactive images: Cuboid proxies for smart image manipulation. ACM Trans. Graph. 31, 4 (July 2012), 99:1–99:11. Google ScholarDigital Library
56. Wei Zhou and Chandra Kambhamettu. 2002. Estimation of illuminant direction and intensity of multiple light sources. In ECCV’02. 206–220. Google ScholarDigital Library
57. Todd Zickler, Ravi Ramamoorthi, Sebastian Enrique, and Peter N. Belhumeur. 2006. Reflectance sharing: Predicting appearance from a sparse set of images of a known shape. IEEE TPAMI 28, 8 (Aug. 2006), 1287–1302. Google ScholarDigital Library
58. Daniel Zoran, Dilip Krishnan, Jose Bento, and Bill Freeman. 2014. Shape and illumination from shading using the generic viewpoint assumption. In NIPS’14. 226–234. Google ScholarDigital Library
