“High dynamic range and super-resolution from raw image bursts” by Lecouat, Eboli, Ponce and Mairal

Photographs captured by smartphones and mid-range cameras have limited spatial resolution and dynamic range, with noisy response in underexposed regions and color artefacts in saturated areas. This paper introduces the first approach (to the best of our knowledge) to the reconstruction of highresolution, high-dynamic range color images from raw photographic bursts captured by a handheld camera with exposure bracketing. This method uses a physically-accurate model of image formation to combine an iterative optimization algorithm for solving the corresponding inverse problem with a learned image representation for robust alignment and a learned natural image prior. The proposed algorithm is fast, with low memory requirements compared to state-of-the-art learning-based approaches to image restoration, and features that are learned end to end from synthetic yet realistic data. Extensive experiments demonstrate its excellent performance with super-resolution factors of up to ×4 on real photographs taken in the wild with hand-held cameras, and high robustness to low-light conditions, noise, camera shake, and moderate object motion.

1. Cecilia Aguerrebere, Julie Delon, Yann Gousseau, and Pablo Musé. 2014. Best Algorithms for HDR Image Generation. A Study of Performance Bounds. SIAM Journal on Imaging Science 7, 1 (2014), 1–34.Google ScholarDigital Library
2. Paul E. Anuta. 1970. Spatial Registration of Multispectral and Multitemporal Digital Imagery Using Fast Fourier Transform Techniques. IEEE Transactions on Geoscience eletronics 8, 4 (1970), 353–368.Google ScholarCross Ref
3. Tunç Ozan Aydin, Rafal Mantiuk, and Hans-Peter Seidel. 2008. Extending quality metrics to full luminance range images. In Proceedings of Human Vision and Electronic Imaging (SPIE Proceedings), Bernice E. Rogowitz and Thrasyvoulos N. Pappas (Eds.), Vol. 6806. SPIE, 68060B.Google Scholar
4. Atilim Gunes Baydin, Barak A Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. 2018. Automatic differentiation in machine learning: a survey. Journal of Machine Learning Research (JMLR) 18 (2018), 1–43.Google Scholar
5. Goutam Bhat, Martin Danelljan, Luc Van Gool, and Radu Timofte. 2021a. Deep Burst Super-Resolution. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR). 9209–9218.Google ScholarCross Ref
6. Goutam Bhat, Martin Danelljan, Fisher Yu, Luc Van Gool, and Radu Timofte. 2021b. Deep Reparametrization of Multi-Frame Super-Resolution and Denoising. (2021), 2460–2470.Google Scholar
7. Tim Brooks, Ben Mildenhall, Tianfan Xue, Jiawen Chen, Dillon Sharlet, and Jonathan T. Barron. 2019. Unprocessing Images for Learned Raw Denoising. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR). 11036–11045.Google Scholar
8. Che-Han Chang, Chun-Nan Chou, and Edward Y Chang. 2017. CLKN: Cascaded lucaskanade networks for image alignment. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR). 2213–2221.Google Scholar
9. Jongseong Choi, Min Kyu Park, and Moon Gi Kang. 2009. High Dynamic Range Image Reconstruction with Spatial Resolution Enhancement. Computer Journal 52, 1 (2009), 114–125.Google ScholarDigital Library
10. Roger N. Clark. 2006. Digital Camera Reviews and Sensor Performance Summary. “https://clarkvision.com/articles/digital.sensor.performance.summary/”.Google Scholar
11. Ryan Dahl, Mohammad Norouzi, and Jonathon Shlens. 2017. Pixel Recursive Super Resolution. In Proceedings of the International Conference on Computer Vision (ICCV).Google ScholarCross Ref
12. Paul E. Debevec and Jitendra Malik. 1997. Recovering high dynamic range radiance maps from photographs. In SIGGRAPH. ACM, 369–378.Google Scholar
13. Xin Deng, Yutong Zhang, Mai Xu, Shuhang Gu, and Yiping Duan. 2021. Deep Coupled Feedback Network for Joint Exposure Fusion and Image Super-Resolution. IEEE Transactions on Image Processing (TIP) 30 (2021), 3098–3112.Google ScholarCross Ref
14. Akshay Dudhane, Syed Waqas Zamir, Salman Khan, Fahad Khan, and Ming-Hsuan Yang. 2021. Burst Image Restoration and Enhancement. arXiv preprint arXiv:2110.03680 (2021).Google Scholar
15. Gabriel Eilertsen, Saghi Hajisharif, Param Hanji, Apostolia Tsirikoglou, Rafal K. Mantiuk, and Jonas Unger. 2021. How to cheat with metrics in single-image HDR reconstruction. In Proceedings of the workshops of the International Conference on Computer Vision (ICCVW). 3981–3990.Google ScholarCross Ref
16. Gabriel Eilertsen, Joel Kronander, Gyorgy Denes, Rafal K. Mantiuk, and Jonas Unger. 2017. HDR image reconstruction from a single exposure using deep CNNs. ACM Transactions on Graphics 36, 6 (2017), 178:1–178:15.Google ScholarDigital Library
17. Yuki Endo, Yoshihiro Kanamori, and Jun Mitani. 2017. Deep reverse tone mapping. ACM Transactions on Graphics (ToG) 36, 6 (2017), 177:1–177:10.Google ScholarDigital Library
18. Manfred Ernst and Bartlomiej Wronski. 2021. HDR+ with Bracketing on Pixel Phones. “https://ai.googleblog.com/2021/04/hdr-with-bracketing-on-pixel-phones.html”.Google Scholar
19. Sina Farsiu, Michael Elad, and Peyman Milanfar. 2006. Multiframe demosaicing and super-resolution of color images. IEEE Transactions on Image Processing (TIP) 15, 1 (2006), 141–159.Google ScholarDigital Library
20. Alessandro Foi, Mejdi Trimeche, Vladimir Katkovnik, and Karen O. Egiazarian. 2008. Practical Poissonian-Gaussian Noise Modeling and Fitting for Single-Image Raw-Data. IEEE Transactions on Image Processing (TIP) 17, 10 (2008), 1737–1754.Google ScholarDigital Library
21. Orazio Gallo, Marius Tico, Roberto Manduchi, Natasha Gelfand, and Kari Pulli. 2012. Metering for Exposure Stacks. Computer Graphics Forum 31, 2 (2012), 479–488.Google ScholarDigital Library
22. Orazio Gallo, Alejandro J. Troccoli, Jun Hu, Kari Pulli, and Jan Kautz. 2015. Locally non-rigid registration for mobile HDR photography. In (CVPRW). IEEE Computer Society, 48–55.Google Scholar
23. Donald Geman and Chengda Yang. 1995. Nonlinear image recovery with half-quadratic regularization. IEEE Transactions on Image Processing (TIP) 5, 7 (1995), 932–946.Google ScholarDigital Library
24. Miguel Granados, Boris Ajdin, Michael Wand, Christian Theobalt, Hans-Peter Seidel, and Hendrik P. A. Lensch. 2010. Optimal HDR reconstruction with linear digital cameras. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR). IEEE Computer Society, 215–222.Google Scholar
25. Bahadir K. Gunturk and Murat Gevrekci. 2006. High-resolution image reconstruction from multiple differently exposed images. IEEE Signal Processing Letters 13, 4 (2006), 197–200.Google ScholarCross Ref
26. Param Hanji, Fangcheng Zhong, and Rafal K. Mantiuk. 2020. Noise-Aware Merging of High Dynamic Range Image Stacks Without Camera Calibration. In Proceedings of the workshops of the European Conference on Computer Vision (ECCVW). 376–391.Google Scholar
27. Samuel W. Hasinoff, Frédo Durand, and William T. Freeman. 2010. Noise-optimal capture for high dynamic range photography. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR). 553–560.Google Scholar
28. Samuel W. Hasinoff, Dillon Sharlet, Ryan Geiss, Andrew Adams, Jonathan T. Barron, Florian Kainz, Jiawen Chen, and Marc Levoy. 2016. Burst photography for high dynamic range and low-light imaging on mobile cameras. ACM Transactions on Graphics (ToG) 35, 6 (2016), 192:1–192:12.Google ScholarDigital Library
29. Felix Heide, Markus Steinberger, Yun-Ta Tsai, Mushfiqur Rouf, Dawid Pajak, Dikpal Reddy, Orazio Gallo, Jing Liu abd Wolfgang Heidrich, Karen Egiazarian, Jan Kautz, and Kari Pulli. 2014. FlexISP: A flexible camera image processing framework. ACM Transactions on Graphics (ToG) 33, 6 (2014), 231:1–231:13.Google ScholarDigital Library
30. Andrey Ignatov, Luc Van Gool, and Radu Timofte. 2020. Replacing mobile camera isp with a single deep learning model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 536–537.Google ScholarCross Ref
31. Nima Khademi Kalantari and Ravi Ramamoorthi. 2017. Deep high dynamic range imaging of dynamic scenes. ACM Transactions on Graphics (ToG) 36, 4 (2017), 144:1–144:12.Google ScholarDigital Library
32. Soo Ye Kim, Jihyong Oh, and Munchurl Kim. 2019. Deep SR-ITM: Joint Learning of Super-Resolution and Inverse Tone-Mapping for 4K UHD HDR Applications. In Proceedings of the International Conference on Computer Vision (ICCV). 3116–3125.Google ScholarCross Ref
33. Bruno Lecouat, Jean Ponce, and Julien Mairal. 2021. Lucas-Kanade Reloaded: End-to-End Super-Resolution from Raw Image Bursts. In Proceedings of the International Conference on Computer Vision (ICCV).Google ScholarCross Ref
34. Anat Levin, Robert Fergus, Frédo Durand, and William T. Freeman. 2007. Image and depth from a conventional camera with a coded aperture. ACM Transactions on Graphics (ToG) 26, 3 (2007), 70.Google ScholarDigital Library
35. Orly Liba, Kiran Murthy, Yun-Ta Tsai, Tim Brooks, Tianfan Xue, Nikhil Karnad, Qiurui He, Jonathan T. Barron, Dillon Sharlet, Ryan Geiss, Samuel W. Hasinoff, Yael Pritch, and Marc Levoy. 2019. Handheld mobile photography in very low light. ACM Transactions on Graphics (ToG) 38, 6 (2019), 164:1–164:16.Google ScholarDigital Library
36. Yu-Lun Liu, Wei-Sheng Lai, Yu-Sheng Chen, Yi-Lung Kao, Ming-Hsuan Yang, Yung-Yu Chuang, and Jia-Bin Huang. 2020. Single-Image HDR Reconstruction by Learning to Reverse the Camera Pipeline. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR). 1648–1657.Google ScholarCross Ref
37. Bruce D. Lucas and Takeo Kanade. 1981. An Iterative Image Registration Technique with an Application to Stereo Vision. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI). 674–679.Google Scholar
38. Guillermo Luijk. 2007. Dcraw tutorial. “http://guillermoluijk.com/tutorial/dcraw/index_en.htm”.Google Scholar
39. Ziwei Luo, Lei Yu, Xuan Mo, Youwei Li, Lanpeng Jia, Haoqiang Fan, Jian Sun, and Shuaicheng Liu. 2021. EBSR: Feature enhanced burst super-resolution with deformable alignment. In Proceedings of the Conference on Computer Vision and Pattern Recognition Workshops. 471–478.Google ScholarCross Ref
40. Kede Ma, Hui Li, Hongwei Yong, Zhou Wang, Deyu Meng, and Lei Zhang. 2017. Robust Multi-Exposure Image Fusion: A Structural Patch Decomposition Approach. IEEE Transactions on Image Processing (TIP) 26, 5 (2017), 2519–2532.Google ScholarDigital Library
41. Henrique S. Malvar, Li-wei He, and Ross Cutler. 2004. High-quality linear interpolation for demosaicing of Bayer-patterned color images. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP). 485–488.Google ScholarCross Ref
42. Steve Mann and Rosalind W. Picard. 1995. On being ‘undigital’ with digital cameras: Extending dynamic range by combining differently exposed pictures. In Proceedings of Is&T. 442–448.Google Scholar
43. Julien NP Martel, Lorenz K Mueller, Stephen J Carey, Piotr Dudek, and Gordon Wetzstein. 2020. Neural sensors: Learning pixel exposures for HDR imaging and video compressive sensing with programmable sensors. IEEE Transactions on Pattern Analysis and Machine Intelligence 42, 7 (2020), 1642–1653.Google ScholarCross Ref
44. Emil Martinec. 2008. Noise, Dynamic Range and Bit Depth in Digital SLRs. “https://photonstophotos.net/EmilMartinec/noise.html”.Google Scholar
45. Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi, and Cynthia Rudin. 2020. PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR).Google ScholarCross Ref
46. Ben Mildenhall, Peter Hedman, Ricardo Martin-Brualla, Pratul Srinivasan, and Jonathan T Barron. 2021. NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Images. arXiv preprint arXiv:2111.13679 (2021).Google Scholar
47. Antoine Monod, Julie Delon, and Thomas Veit. 2021. An Analysis and Implementation of the HDR+ Burst Denoising Method. Image Processing On Line 11 (2021), 142–169.Google ScholarCross Ref
48. Manish Narwaria, Rafal K. Mantiuk, Matthieu Perreira Da Silva, and Patrick Le Callet. 2015. HDR-VDP-2.2: a calibrated method for objective quality prediction of high-dynamic range and standard images. Journal on Electronic Imaging 24, 1 (2015), 010501.Google ScholarCross Ref
49. Shree K Nayar and Tomoo Mitsunaga. 2000. High dynamic range imaging: Spatially varying pixel exposures. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), Vol. 1. IEEE, 472–479.Google ScholarCross Ref
50. Yuzhen Niu, Jianbin Wu, Wenxi Liu, Wenzhong Guo, and Rynson W. H. Lau. 2021. HDR-GAN: HDR Image Reconstruction From Multi-Exposed LDR Images With Large Motions. IEEE Transactions on Image Processing (TIP) 30 (2021), 3885–3896.Google ScholarCross Ref
51. Neal Parikh and Stephen P. Boyd. 2014. Proximal Algorithms. Foundations and Trends in Optimization 1, 3 (2014), 127–239.Google ScholarDigital Library
52. Eduardo Pérez-Pellitero, Sibi Catley-Chandar, Ales Leonardis, and Radu Timofte. 2021. NTIRE 2021 Challenge on High Dynamic Range Imaging: Dataset, Methods and Results. In CVPR Workshops. 691–700.Google Scholar
53. Tobias Plötz and Stefan Roth. 2017. Benchmarking Denoising Algorithms with Real Photographs. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR). 2750–2759.Google ScholarCross Ref
54. Ali Ajdari Rad, Laurence Meylan, Patrick Vandewalle, and Sabine Süsstrunk. 2007. Multidimensional image enhancement from a set of unregistered and differently exposed images. In Computational Imaging (SPIE Proceedings), Vol. 6498. SPIE, 649808.Google Scholar
55. Erik Reinhard, Michael M. Stark, Peter Shirley, and James A. Ferwerda. 2002. Photographic tone reproduction for digital images. ACM Transactions on Graphics (ToG) 21, 3 (2002), 267–276.Google ScholarDigital Library
56. Javier Sanchez. 2016. The inverse compositional algorithm for parametric registration. Image Processing On Line (2016).Google Scholar
57. Marcel Santana Santos, Tsang Ing Ren, and Nima Khademi Kalantari. 2020. Single image HDR reconstruction using a CNN with masked features and perceptual loss. ACM Transactions on Graphics (ToG) 39, 4 (2020), 80.Google ScholarDigital Library
58. Pradeep Sen, Nima Khademi Kalantari, Maziar Yaesoubi, Soheil Darabi, Dan B. Goldman, and Eli Shechtman. 2012. Robust patch-based HDR reconstruction of dynamic scenes. ACM Transactions on Graphics (ToG) 31, 6 (2012), 203:1–203:11.Google ScholarDigital Library
59. Ana Serrano, Felix Heide, Diego Gutierrez, Gordon Wetzstein, and Belen Masia. 2016. Convolutional sparse coding for high dynamic range imaging. In Computer Graphics Forum, Vol. 35. Wiley Online Library, 153–163.Google Scholar
60. Hiroyuki Takeda, Sina Farsiu, and Peyman Milanfar. 2007. Kernel Regression for Image Processing and Reconstruction. IEEE Transactions on Image Processing (TIP) 16, 2 (2007), 349–366.Google ScholarDigital Library
61. Yann Traonmilin and Cecilia Aguerrebere. 2014. Simultaneous High Dynamic Range and Superresolution Imaging without Regularization. SIAM Journal on Imaging Science 7, 3 (2014), 1624–1644.Google ScholarDigital Library
62. Okan Tarhan Tursun, Ahmet Oguz Akyüz, Aykut Erdem, and Erkut Erdem. 2016. An Objective Deghosting Quality Metric for HDR Images. Computer Graphics Forum 35, 2 (2016), 139–152.Google ScholarCross Ref
63. Patrick Vandewalle, Sabine Süsstrunk, and Martin Vetterli. 2006. A Frequency Domain Approach to Registration of Aliased Images with Application to Super-resolution. EURASIP Journal on Advances in Signal Processing 2006 (2006).Google Scholar
64. Subeesh Vasu, Abhijeet Shenoi, and A. N. Rajagopalan. 2018. Joint HDR and SuperResolution Imaging in Motion Blur. In Proceedings of the International Conference on Image Processing (ICIP). 2885–2889.Google Scholar
65. Singanallur V Venkatakrishnan, Charles A Bouman, and Brendt Wohlberg. 2013. Plug-and-play priors for model based reconstruction. In Proceedings of the Global Conference on Signal and Information Processing. 945–948.Google ScholarCross Ref
66. Greg Ward. 2003. Fast, Robust Image Registration for Compositing High Dynamic Range Photographs from Hand-Held Exposures. Journal on Graphics, GPU, & Game Tools 8, 2 (2003), 17–30.Google ScholarCross Ref
67. Bartlomiej Wronski, Ignacio Garcia-Dorado, Manfred Ernst, Damien Kelly, Michael Krainin, Chia-Kai Liang, Marc Levoy, and Peyman Milanfar. 2019. Handheld multiframe super-resolution. ACM Transactions on Graphics (ToG) 38, 4 (2019), 28:1–28:18.Google ScholarDigital Library
68. Shangzhe Wu, Jiarui Xu, Yu-Wing Tai, and Chi-Keung Tang. 2018. Deep High Dynamic Range Imaging with Large Foreground Motions. In Proceedings of the European Conference on Computer Vision (ECCV). 120–135.Google ScholarDigital Library
69. Qingsen Yan, Dong Gong, Javen Qinfeng Shi, Anton van den Hengel, Chunhua Shen, Ian Reid, and Yanning Zhang. 2021. Dual-attention-guided network for ghost-free high dynamic range imaging. International Journal of Computer Vision (IJCV) (2021), 1–19.Google Scholar
70. Qingsen Yan, Lei Zhang, Yu Liu, Yu Zhu, Jinqiu Sun, Qinfeng Shi, and Yanning Zhang. 2020. Deep HDR Imaging via A Non-Local Network. IEEE Transactions on Image Processing (TIP) 29 (2020), 4308–4322.Google ScholarCross Ref
71. Kai Zhang, Luc Van Gool, and Radu Timofte. 2020. Deep unfolding network for image super-resolution. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR). 3217–3226.Google ScholarCross Ref
72. Henning Zimmer, Andrés Bruhn, and Joachim Weickert. 2011. Freehand HDR Imaging of Moving Scenes with Simultaneous Resolution Enhancement. Computer Graphics Forum 30, 2 (2011), 405–414.Google ScholarCross Ref