“Deep high dynamic range imaging of dynamic scenes” by Kalantari and Ramamoorthi
Conference:
Type(s):
Title:
- Deep high dynamic range imaging of dynamic scenes
Session/Category Title: Image and Light Field Manipulation
Presenter(s)/Author(s):
Moderator(s):
Abstract:
Producing a high dynamic range (HDR) image from a set of images with different exposures is a challenging process for dynamic scenes. A category of existing techniques first register the input images to a reference image and then merge the aligned images into an HDR image. However, the artifacts of the registration usually appear as ghosting and tearing in the final HDR images. In this paper, we propose a learning-based approach to address this problem for dynamic scenes. We use a convolutional neural network (CNN) as our learning model and present and compare three different system architectures to model the HDR merge process. Furthermore, we create a large dataset of input LDR images and their corresponding ground truth HDR images to train our system. We demonstrate the performance of our system by producing high-quality HDR images from a set of three LDR images. Experimental results show that our method consistently produces better results than several state-of-the-art approaches on challenging scenes.
References:
1. A. Badki, N. Khademi Kalantari, and P. Sen. 2015. Robust Radiometric Calibration for Dynamic Scenes in the Wild. In IEEE ICCP. 1–10. Google ScholarCross Ref
2. Luca Bogoni. 2000. Extending Dynamic Range of Monochrome and Color Images through Fusion. In IEEE ICPR. 3007–3016. Google ScholarCross Ref
3. Z. Chen, H. Jin, Z. Lin, S. Cohen, and Y. Wu. 2013. Large Displacement Optical Flow from Nearest Neighbor Fields. In IEEE CVPR. 2443–2450. Google ScholarDigital Library
4. Z. Cheng, Q. Yang, and B. Sheng. 2015. Deep Colorization. In IEEE ICCV. 415–423. Google ScholarDigital Library
5. Paul E. Debevec and Jitendra Malik. 1997. Recovering high dynamic range radiance maps from photographs. In ACM SIGGRAPH. 369–378.Google Scholar
6. A. Dosovitskiy, P. Fischery, E. Ilg, P. HÃd’usser, C. Hazirbas, V. Golkov, P. v. d. Smagt, D. Cremers, and T. Brox. 2015. FlowNet: Learning Optical Flow with Convolutional Networks. In ICCV. 2758–2766.Google Scholar
7. John Flynn, Ivan Neulander, James Philbin, and Noah Snavely. 2016. DeepStereo: Learning to Predict New Views from the WorldâĂŹs Imagery. In IEEE CVPR. 5515–5524.Google Scholar
8. Jan Froehlich, Stefan Grandinetti, Bernd Eberhardt, Simon Walter, Andreas Schilling, and Harald Brendel. 2014. Creating cinematic wide gamut HDR-video for the evaluation of tone mapping operators and HDR-displays. SPIE 9023 (2014), 90230X–90230X-10.Google Scholar
9. Brian Funt and Lilong Shi. 2010. The Rehabilitation of MaxRGB. Color and Imaging Conference 2010, 1 (2010), 256–259.Google Scholar
10. O. Gallo, N. Gelfand, W. Chen, M. Tico, and K. Pulli. 2009. Artifact-free High Dynamic Range Imaging. In IEEE ICCP. 1–7. Google ScholarCross Ref
11. O. Gallo, A. Troccoli, J. Hu, K. Pulli, and J. Kautz. 2015. Locally non-rigid registration for mobile HDR photography. In IEEE CVPRW. 48–55. Google ScholarCross Ref
12. Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In AISTATS, Vol. 9. 249–256.Google Scholar
13. I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. 2014. Generative Adversarial Networks. ArXiv e-prints (June 2014). arXiv:stat.ML/1406.2661Google Scholar
14. M. Granados, B. Ajdin, M. Wand, C. Theobalt, H. P. Seidel, and H. P. A. Lensch. 2010. Optimal HDR reconstruction with linear digital cameras. In IEEE CVPR. 215–222. Google ScholarCross Ref
15. Miguel Granados, Kwang In Kim, James Tompkin, and Christian Theobalt. 2013. Automatic Noise Modeling for Ghost-free HDR Reconstruction. ACM TOG 32, 6, Article 201 (2013), 10 pages.Google Scholar
16. T. Grosch. 2006. Fast and Robust High Dynamic Range Image Generation with Camera and Object Movement. In Vision, Modeling and Visualization. 277–284.Google Scholar
17. Michael D. Grossberg and Shree K. Nayar. 2003. Determining the Camera Response from Images: What Is Knowable? IEEE PAMI 25, 11 (Nov. 2003), 1455–1467. Google ScholarDigital Library
18. Yoav HaCohen, Eli Shechtman, Dan B. Goldman, and Dani Lischinski. 2011. Non-rigid dense correspondence with applications for image enhancement. ACM TOG 30, 4, Article 70 (2011), 10 pages.Google Scholar
19. Saghi Hajisharif, Joel Kronander, and Jonas Unger. 2015. Adaptive dualISO HDR reconstruction. EURASIP Journal on Image and Video Processing 2015, 1 (2015), 41.Google ScholarCross Ref
20. S. W. Hasinoff, F. Durand, and W. T. Freeman. 2010. Noise-optimal capture for high dynamic range photography. In CVPR. 553–560. Google ScholarCross Ref
21. Samuel W. Hasinoff, Dillon Sharlet, Ryan Geiss, Andrew Adams, Jonathan T. Barron, Florian Kainz, Jiawen Chen, and Marc Levoy. 2016. Burst Photography for High Dynamic Range and Low-light Imaging on Mobile Cameras. ACM TOG 35, 6, Article 192 (Nov. 2016), 12 pages.Google Scholar
22. Felix Heide, Markus Steinberger, Yun-Ta Tsai, Mushfiqur Rouf, Dawid Pająk, Dikpal Reddy, Orazio Gallo, Jing Liu, Wolfgang Heidrich, Karen Egiazarian, Jan Kautz, and Kari Pulli. 2014. FlexISP: A Flexible Camera Image Processing Framework. ACM TOG 33, 6, Article 231 (Nov. 2014), 13 pages.Google Scholar
23. Yong Seok Heo, Kyoung Mu Lee, Sang Uk Lee, Youngsu Moon, and Joonhyuk Cha. 2010. Ghost-free high dynamic range imaging. In ACCV, Vol. 4. 486–500.Google Scholar
24. Jun Hu, Orazio Gallo, and Kari Pulli. 2012. Exposure Stacks of Live Scenes with Hand-Held Cameras. Springer Berlin Heidelberg, Berlin, Heidelberg, 499–512.Google Scholar
25. J. Hu, O. Gallo, K. Pulli, and X. Sun. 2013. HDR Deghosting: How to Deal with Saturation?. In IEEE CVPR. 1163–1170.Google Scholar
26. Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. 2016. Let There Be Color!: Joint End-to-end Learning of Global and Local Image Priors for Automatic Image Colorization with Simultaneous Classification. ACM TOG 35, 4, Article 110 (July 2016), 11 pages.Google Scholar
27. Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox. 2016. FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks. CoRR abs/1612.01925 (2016).Google Scholar
28. K. Jacobs, C. Loscos, and G. Ward. 2008. Automatic High-Dynamic Range Image Generation for Dynamic Scenes. IEEE Computer Graphics and Applications 28, 2 (Mar.-Apr. 2008), 84–93. Google ScholarDigital Library
29. T. Jinno and M. Okuda. 2008. Motion blur free HDR image acquisition using multiple exposures. In IEEE ICIP. 1304–1307. Google ScholarCross Ref
30. Nima Khademi Kalantari, Ting-Chun Wang, and Ravi Ramamoorthi. 2016. Learning-based View Synthesis for Light Field Cameras. ACM TOG 35, 6, Article 193 (Nov. 2016), 10 pages.Google Scholar
31. Sing Bing Kang, Matthew Uyttendaele, Simon Winder, and Richard Szeliski. 2003. High dynamic range video. ACM TOG 22, 3 (2003), 319–325. Google ScholarDigital Library
32. Kanita Karaduzovic-Hadziabdic, Jasminka Hasic Telalovic, and Rafal Mantiuk. 2016. Subjective and Objective Evaluation of Multi-exposure High Dynamic Range Image Deghosting Methods. In EG, T. Bashford-Rogers and L. P. Santos (Eds.). The Eurographics Association. Google ScholarCross Ref
33. E.A. Khan, A.O. Akyüz, and E. Reinhard. 2006. Ghost Removal in High Dynamic Range Images. In IEEE ICIP. 2005–2008. Google ScholarCross Ref
34. C. Lee, Y. Li, and V. Monga. 2014. Ghost-Free High Dynamic Range Imaging via Rank Minimization. IEEE SPL 21, 9 (Sept 2014), 1045–1049.Google Scholar
35. Z. Li, J. Zheng, Z. Zhu, and S. Wu. 2014. Selectively Detail-Enhanced Fusion of Differently Exposed Images With Moving Objects. IEEE Transactions on Image Processing 23, 10 (Oct 2014), 4372–4382. Google ScholarCross Ref
36. C. Liu. 2009. Beyond Pixels: Exploring New Representations and Applications for Motion Analysis. Doctoral thesis. Massachusetts Institute of Technology.Google Scholar
37. Bruce D. Lucas and Takeo Kanade. 1981. An Iterative Image Registration Technique with an Application to Stereo Vision. In IJCAI. 674–679.Google Scholar
38. Steve Mann and Rosalind W. Picard. 1995. On Being ‘undigital’ With Digital Cameras: Extending Dynamic Range By Combining Differently Exposed Pictures. In Proc. of Society for Imaging Science and Technology. 442–448.Google Scholar
39. Rafat Mantiuk, Kil Joong Kim, Allan G. Rempel, and Wolfgang Heidrich. 2011. HDR-VDP-2: A Calibrated Visual Metric for Visibility and Quality Predictions in All Luminance Conditions. ACM TOG 30, 4, Article 40 (July 2011), 14 pages.Google Scholar
40. M. McGuire, W. Matusik, H. Pfister, B. Chen, J.F. Hughes, and S.K. Nayar. 2007. Optical Splitting Trees for High-Precision Monocular Imaging. IEEE Computer Graphics and Applications 27, 2 (march-april 2007), 32–42. Google ScholarDigital Library
41. T. H. Oh, J. Y. Lee, Y. W. Tai, and I. S. Kweon. 2015. Robust High Dynamic Range Imaging by Rank Minimization. IEEE PAMI 37, 6 (2015), 1219–1232. Google ScholarCross Ref
42. F. Pece and J. Kautz. 2010. Bitmap Movement Detection: HDR for Dynamic Scenes. In CVMP. 1–8.Google Scholar
43. Photomatix. 2017. Commercially-available HDR processing software. (2017). http://www.hdrsoft.com/.Google Scholar
44. Shanmuganathan Raman and Subhasis Chaudhuri. 2011. Reconstruction of high contrast images for dynamic scenes. The Visual Computer 27, 12 (Dec. 2011), 1099–1114. Google ScholarDigital Library
45. David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. 1986. Learning Representations by Back-propagating Errors. Nature 323 (1986), 533–536. Google ScholarCross Ref
46. Pradeep Sen, Nima Khademi Kalantari, Maziar Yaesoubi, Soheil Darabi, Dan B. Goldman, and Eli Shechtman. 2012. Robust patch-based HDR reconstruction of dynamic scenes. ACM TOG 31, 6, Article 203 (Nov. 2012), 11 pages.Google Scholar
47. Ana Serrano, Felix Heide, Diego Gutierrez, Gordon Wetzstein, and Belen Masia. 2016. Convolutional Sparse Coding for High Dynamic Range Imaging. CGF 35, 2 (2016), 153–163. Google ScholarDigital Library
48. Michael D. Tocci, Chris Kiser, Nora Tocci, and Pradeep Sen. 2011. A versatile HDR video production system. ACM TOG 30, 4, Article 41 (July 2011), 10 pages.Google Scholar
49. A. Tomaszewska and R. Mantiuk. 2007. Image Registration for Multi-exposure High Dynamic Range Image Acquisition. In WSCG.Google Scholar
50. Okan Tarhan Tursun, Ahmet Oğuz Akyüz, Aykut Erdem, and Erkut Erdem. 2016. An Objective Deghosting Quality Metric for HDR Images. CGF 35, 2 (2016), 139–152. Google ScholarDigital Library
51. Okan Tarhan Tursun, Ahmet Oħuz AkyAijz, Aykut Erdem, and Erkut Erdem. 2015. The State of the Art in HDR Deghosting: A Survey and Evaluation. CGF 34, 2 (2015).Google Scholar
52. Andrea Vedaldi and Karel Lenc. 2015. MatConvNet: Convolutional neural networks for Matlab. In ACMMM. 689–692.Google Scholar
53. Greg Ward. 2003. Fast, Robust Image Registration for Compositing High Dynamic Range Photographs from Hand-Held Exposures. Journal of Graphics, GPU, and Game Tools 8, 2 (2003), 17–30. Google ScholarCross Ref
54. L. Zhang, A. Deshpande, and X. Chen. 2010. Denoising vs. deblurring: HDR imaging techniques using moving cameras. In CVPR. 522–529.Google Scholar
55. Wei Zhang and Wai-Kuen Cham. 2012. Gradient-Directed Multiexposure Composition. IEEE TIP 21, 4 (April 2012), 2318–2323. Google ScholarDigital Library
56. H. Zhao, B. Shi, C. Fernandez-Cull, S. K. Yeung, and R. Raskar. 2015. Unbounded High Dynamic Range Photography Using a Modulo Camera. In ICCP 2015. 1–10. Google ScholarCross Ref
57. J. Zheng, Z. Li, Z. Zhu, S. Wu, and S. Rahardja. 2013. Hybrid Patching for a Sequence of Differently Exposed Images With Moving Objects. IEEE TIP 22, 12 (Dec 2013), 5190–5201. Google ScholarDigital Library
58. Henning Zimmer, AndrÃl’s Bruhn, and Joachim Weickert. 2011. Freehand HDR Imaging of Moving Scenes with Simultaneous Resolution Enhancement. CGF 30, 2 (April 2011), 405–414.Google ScholarCross Ref
