“Deep high dynamic range imaging of dynamic scenes” by Kalantari and Ramamoorthi

  • ©Nima Khademi Kalantari and Ravi Ramamoorthi




    Deep high dynamic range imaging of dynamic scenes

Session/Category Title: Image and Light Field Manipulation




    Producing a high dynamic range (HDR) image from a set of images with different exposures is a challenging process for dynamic scenes. A category of existing techniques first register the input images to a reference image and then merge the aligned images into an HDR image. However, the artifacts of the registration usually appear as ghosting and tearing in the final HDR images. In this paper, we propose a learning-based approach to address this problem for dynamic scenes. We use a convolutional neural network (CNN) as our learning model and present and compare three different system architectures to model the HDR merge process. Furthermore, we create a large dataset of input LDR images and their corresponding ground truth HDR images to train our system. We demonstrate the performance of our system by producing high-quality HDR images from a set of three LDR images. Experimental results show that our method consistently produces better results than several state-of-the-art approaches on challenging scenes.


    1. A. Badki, N. Khademi Kalantari, and P. Sen. 2015. Robust Radiometric Calibration for Dynamic Scenes in the Wild. In IEEE ICCP. 1–10. Google ScholarCross Ref
    2. Luca Bogoni. 2000. Extending Dynamic Range of Monochrome and Color Images through Fusion. In IEEE ICPR. 3007–3016. Google ScholarCross Ref
    3. Z. Chen, H. Jin, Z. Lin, S. Cohen, and Y. Wu. 2013. Large Displacement Optical Flow from Nearest Neighbor Fields. In IEEE CVPR. 2443–2450. Google ScholarDigital Library
    4. Z. Cheng, Q. Yang, and B. Sheng. 2015. Deep Colorization. In IEEE ICCV. 415–423. Google ScholarDigital Library
    5. Paul E. Debevec and Jitendra Malik. 1997. Recovering high dynamic range radiance maps from photographs. In ACM SIGGRAPH. 369–378.Google Scholar
    6. A. Dosovitskiy, P. Fischery, E. Ilg, P. HÃd’usser, C. Hazirbas, V. Golkov, P. v. d. Smagt, D. Cremers, and T. Brox. 2015. FlowNet: Learning Optical Flow with Convolutional Networks. In ICCV. 2758–2766.Google Scholar
    7. John Flynn, Ivan Neulander, James Philbin, and Noah Snavely. 2016. DeepStereo: Learning to Predict New Views from the WorldâĂŹs Imagery. In IEEE CVPR. 5515–5524.Google Scholar
    8. Jan Froehlich, Stefan Grandinetti, Bernd Eberhardt, Simon Walter, Andreas Schilling, and Harald Brendel. 2014. Creating cinematic wide gamut HDR-video for the evaluation of tone mapping operators and HDR-displays. SPIE 9023 (2014), 90230X–90230X-10.Google Scholar
    9. Brian Funt and Lilong Shi. 2010. The Rehabilitation of MaxRGB. Color and Imaging Conference 2010, 1 (2010), 256–259.Google Scholar
    10. O. Gallo, N. Gelfand, W. Chen, M. Tico, and K. Pulli. 2009. Artifact-free High Dynamic Range Imaging. In IEEE ICCP. 1–7. Google ScholarCross Ref
    11. O. Gallo, A. Troccoli, J. Hu, K. Pulli, and J. Kautz. 2015. Locally non-rigid registration for mobile HDR photography. In IEEE CVPRW. 48–55. Google ScholarCross Ref
    12. Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In AISTATS, Vol. 9. 249–256.Google Scholar
    13. I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. 2014. Generative Adversarial Networks. ArXiv e-prints (June 2014). arXiv:stat.ML/1406.2661Google Scholar
    14. M. Granados, B. Ajdin, M. Wand, C. Theobalt, H. P. Seidel, and H. P. A. Lensch. 2010. Optimal HDR reconstruction with linear digital cameras. In IEEE CVPR. 215–222. Google ScholarCross Ref
    15. Miguel Granados, Kwang In Kim, James Tompkin, and Christian Theobalt. 2013. Automatic Noise Modeling for Ghost-free HDR Reconstruction. ACM TOG 32, 6, Article 201 (2013), 10 pages.Google Scholar
    16. T. Grosch. 2006. Fast and Robust High Dynamic Range Image Generation with Camera and Object Movement. In Vision, Modeling and Visualization. 277–284.Google Scholar
    17. Michael D. Grossberg and Shree K. Nayar. 2003. Determining the Camera Response from Images: What Is Knowable? IEEE PAMI 25, 11 (Nov. 2003), 1455–1467. Google ScholarDigital Library
    18. Yoav HaCohen, Eli Shechtman, Dan B. Goldman, and Dani Lischinski. 2011. Non-rigid dense correspondence with applications for image enhancement. ACM TOG 30, 4, Article 70 (2011), 10 pages.Google Scholar
    19. Saghi Hajisharif, Joel Kronander, and Jonas Unger. 2015. Adaptive dualISO HDR reconstruction. EURASIP Journal on Image and Video Processing 2015, 1 (2015), 41.Google ScholarCross Ref
    20. S. W. Hasinoff, F. Durand, and W. T. Freeman. 2010. Noise-optimal capture for high dynamic range photography. In CVPR. 553–560. Google ScholarCross Ref
    21. Samuel W. Hasinoff, Dillon Sharlet, Ryan Geiss, Andrew Adams, Jonathan T. Barron, Florian Kainz, Jiawen Chen, and Marc Levoy. 2016. Burst Photography for High Dynamic Range and Low-light Imaging on Mobile Cameras. ACM TOG 35, 6, Article 192 (Nov. 2016), 12 pages.Google Scholar
    22. Felix Heide, Markus Steinberger, Yun-Ta Tsai, Mushfiqur Rouf, Dawid Pająk, Dikpal Reddy, Orazio Gallo, Jing Liu, Wolfgang Heidrich, Karen Egiazarian, Jan Kautz, and Kari Pulli. 2014. FlexISP: A Flexible Camera Image Processing Framework. ACM TOG 33, 6, Article 231 (Nov. 2014), 13 pages.Google Scholar
    23. Yong Seok Heo, Kyoung Mu Lee, Sang Uk Lee, Youngsu Moon, and Joonhyuk Cha. 2010. Ghost-free high dynamic range imaging. In ACCV, Vol. 4. 486–500.Google Scholar
    24. Jun Hu, Orazio Gallo, and Kari Pulli. 2012. Exposure Stacks of Live Scenes with Hand-Held Cameras. Springer Berlin Heidelberg, Berlin, Heidelberg, 499–512.Google Scholar
    25. J. Hu, O. Gallo, K. Pulli, and X. Sun. 2013. HDR Deghosting: How to Deal with Saturation?. In IEEE CVPR. 1163–1170.Google Scholar
    26. Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. 2016. Let There Be Color!: Joint End-to-end Learning of Global and Local Image Priors for Automatic Image Colorization with Simultaneous Classification. ACM TOG 35, 4, Article 110 (July 2016), 11 pages.Google Scholar
    27. Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox. 2016. FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks. CoRR abs/1612.01925 (2016).Google Scholar
    28. K. Jacobs, C. Loscos, and G. Ward. 2008. Automatic High-Dynamic Range Image Generation for Dynamic Scenes. IEEE Computer Graphics and Applications 28, 2 (Mar.-Apr. 2008), 84–93. Google ScholarDigital Library
    29. T. Jinno and M. Okuda. 2008. Motion blur free HDR image acquisition using multiple exposures. In IEEE ICIP. 1304–1307. Google ScholarCross Ref
    30. Nima Khademi Kalantari, Ting-Chun Wang, and Ravi Ramamoorthi. 2016. Learning-based View Synthesis for Light Field Cameras. ACM TOG 35, 6, Article 193 (Nov. 2016), 10 pages.Google Scholar
    31. Sing Bing Kang, Matthew Uyttendaele, Simon Winder, and Richard Szeliski. 2003. High dynamic range video. ACM TOG 22, 3 (2003), 319–325. Google ScholarDigital Library
    32. Kanita Karaduzovic-Hadziabdic, Jasminka Hasic Telalovic, and Rafal Mantiuk. 2016. Subjective and Objective Evaluation of Multi-exposure High Dynamic Range Image Deghosting Methods. In EG, T. Bashford-Rogers and L. P. Santos (Eds.). The Eurographics Association. Google ScholarCross Ref
    33. E.A. Khan, A.O. Akyüz, and E. Reinhard. 2006. Ghost Removal in High Dynamic Range Images. In IEEE ICIP. 2005–2008. Google ScholarCross Ref
    34. C. Lee, Y. Li, and V. Monga. 2014. Ghost-Free High Dynamic Range Imaging via Rank Minimization. IEEE SPL 21, 9 (Sept 2014), 1045–1049.Google Scholar
    35. Z. Li, J. Zheng, Z. Zhu, and S. Wu. 2014. Selectively Detail-Enhanced Fusion of Differently Exposed Images With Moving Objects. IEEE Transactions on Image Processing 23, 10 (Oct 2014), 4372–4382. Google ScholarCross Ref
    36. C. Liu. 2009. Beyond Pixels: Exploring New Representations and Applications for Motion Analysis. Doctoral thesis. Massachusetts Institute of Technology.Google Scholar
    37. Bruce D. Lucas and Takeo Kanade. 1981. An Iterative Image Registration Technique with an Application to Stereo Vision. In IJCAI. 674–679.Google Scholar
    38. Steve Mann and Rosalind W. Picard. 1995. On Being ‘undigital’ With Digital Cameras: Extending Dynamic Range By Combining Differently Exposed Pictures. In Proc. of Society for Imaging Science and Technology. 442–448.Google Scholar
    39. Rafat Mantiuk, Kil Joong Kim, Allan G. Rempel, and Wolfgang Heidrich. 2011. HDR-VDP-2: A Calibrated Visual Metric for Visibility and Quality Predictions in All Luminance Conditions. ACM TOG 30, 4, Article 40 (July 2011), 14 pages.Google Scholar
    40. M. McGuire, W. Matusik, H. Pfister, B. Chen, J.F. Hughes, and S.K. Nayar. 2007. Optical Splitting Trees for High-Precision Monocular Imaging. IEEE Computer Graphics and Applications 27, 2 (march-april 2007), 32–42. Google ScholarDigital Library
    41. T. H. Oh, J. Y. Lee, Y. W. Tai, and I. S. Kweon. 2015. Robust High Dynamic Range Imaging by Rank Minimization. IEEE PAMI 37, 6 (2015), 1219–1232. Google ScholarCross Ref
    42. F. Pece and J. Kautz. 2010. Bitmap Movement Detection: HDR for Dynamic Scenes. In CVMP. 1–8.Google Scholar
    43. Photomatix. 2017. Commercially-available HDR processing software. (2017). http://www.hdrsoft.com/.Google Scholar
    44. Shanmuganathan Raman and Subhasis Chaudhuri. 2011. Reconstruction of high contrast images for dynamic scenes. The Visual Computer 27, 12 (Dec. 2011), 1099–1114. Google ScholarDigital Library
    45. David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. 1986. Learning Representations by Back-propagating Errors. Nature 323 (1986), 533–536. Google ScholarCross Ref
    46. Pradeep Sen, Nima Khademi Kalantari, Maziar Yaesoubi, Soheil Darabi, Dan B. Goldman, and Eli Shechtman. 2012. Robust patch-based HDR reconstruction of dynamic scenes. ACM TOG 31, 6, Article 203 (Nov. 2012), 11 pages.Google Scholar
    47. Ana Serrano, Felix Heide, Diego Gutierrez, Gordon Wetzstein, and Belen Masia. 2016. Convolutional Sparse Coding for High Dynamic Range Imaging. CGF 35, 2 (2016), 153–163. Google ScholarDigital Library
    48. Michael D. Tocci, Chris Kiser, Nora Tocci, and Pradeep Sen. 2011. A versatile HDR video production system. ACM TOG 30, 4, Article 41 (July 2011), 10 pages.Google Scholar
    49. A. Tomaszewska and R. Mantiuk. 2007. Image Registration for Multi-exposure High Dynamic Range Image Acquisition. In WSCG.Google Scholar
    50. Okan Tarhan Tursun, Ahmet Oğuz Akyüz, Aykut Erdem, and Erkut Erdem. 2016. An Objective Deghosting Quality Metric for HDR Images. CGF 35, 2 (2016), 139–152. Google ScholarDigital Library
    51. Okan Tarhan Tursun, Ahmet Oħuz AkyAijz, Aykut Erdem, and Erkut Erdem. 2015. The State of the Art in HDR Deghosting: A Survey and Evaluation. CGF 34, 2 (2015).Google Scholar
    52. Andrea Vedaldi and Karel Lenc. 2015. MatConvNet: Convolutional neural networks for Matlab. In ACMMM. 689–692.Google Scholar
    53. Greg Ward. 2003. Fast, Robust Image Registration for Compositing High Dynamic Range Photographs from Hand-Held Exposures. Journal of Graphics, GPU, and Game Tools 8, 2 (2003), 17–30. Google ScholarCross Ref
    54. L. Zhang, A. Deshpande, and X. Chen. 2010. Denoising vs. deblurring: HDR imaging techniques using moving cameras. In CVPR. 522–529.Google Scholar
    55. Wei Zhang and Wai-Kuen Cham. 2012. Gradient-Directed Multiexposure Composition. IEEE TIP 21, 4 (April 2012), 2318–2323. Google ScholarDigital Library
    56. H. Zhao, B. Shi, C. Fernandez-Cull, S. K. Yeung, and R. Raskar. 2015. Unbounded High Dynamic Range Photography Using a Modulo Camera. In ICCP 2015. 1–10. Google ScholarCross Ref
    57. J. Zheng, Z. Li, Z. Zhu, S. Wu, and S. Rahardja. 2013. Hybrid Patching for a Sequence of Differently Exposed Images With Moving Objects. IEEE TIP 22, 12 (Dec 2013), 5190–5201. Google ScholarDigital Library
    58. Henning Zimmer, AndrÃl’s Bruhn, and Joachim Weickert. 2011. Freehand HDR Imaging of Moving Scenes with Simultaneous Resolution Enhancement. CGF 30, 2 (April 2011), 405–414.Google ScholarCross Ref

ACM Digital Library Publication:

Overview Page: