“Handheld multi-frame super-resolution” by Wronski, Garcia-Dorado, Ernst, Kelly, Krainin, et al. …
Conference:
Type(s):
Title:
- Handheld multi-frame super-resolution
Session/Category Title: Image Science
Presenter(s)/Author(s):
Abstract:
Compared to DSLR cameras, smartphone cameras have smaller sensors, which limits their spatial resolution; smaller apertures, which limits their light gathering ability; and smaller pixels, which reduces their signal-to-noise ratio. The use of color filter arrays (CFAs) requires demosaicing, which further degrades resolution. In this paper, we supplant the use of traditional demosaicing in single-frame and burst photography pipelines with a multiframe super-resolution algorithm that creates a complete RGB image directly from a burst of CFA raw images. We harness natural hand tremor, typical in handheld photography, to acquire a burst of raw frames with small offsets. These frames are then aligned and merged to form a single image with red, green, and blue values at every pixel site. This approach, which includes no explicit demosaicing step, serves to both increase image resolution and boost signal to noise ratio. Our algorithm is robust to challenging scene conditions: local motion, occlusion, or scene changes. It runs at 100 milliseconds per 12-megapixel RAW input burst frame on mass-produced mobile phones. Specifically, the algorithm is the basis of the Super-Res Zoom feature, as well as the default merge method in Night Sight mode (whether zooming or not) on Google’s flagship phone.
References:
1. Adobe. 2012. Digital Negative (DNG) Specification. https://www.adobe.com/content/dam/acom/en/products/photoshop/pdfs/dng_spec_1.4.0.0.pdf.Google Scholar
2. Simon Baker and Takeo Kanade. 2002. Limits on super-resolution and how to break them. IEEE Trans. PAMI 24, 9 (2002), 1167–1183. Google ScholarDigital Library
3. Bryce E Bayer. 1976. Color imaging array. US Patent 3,971,065.Google Scholar
4. Stefanos P Belekos, Nikolaos P Galatsanos, and Aggelos K Katsaggelos. 2010. Maximum a posteriori video super-resolution using a new multichannel image prior. IEEE Trans. Image Processing 19, 6 (2010), 1451–1464. Google ScholarDigital Library
5. Moshe Ben-Ezra, Assaf Zomet, and Shree K Nayar. 2005. Video super-resolution using controlled subpixel detector shifts. IEEE Trans. PAMI 27, 6 (2005), 977–987. Google ScholarDigital Library
6. Josef Bigün, Goesta H. Granlund, and Johan Wiklund. 1991. Multidimensional orientation estimation with applications to texture analysis and optical flow. IEEE Trans. PAMI 8 (1991), 775–790. Google ScholarDigital Library
7. James F Blinn. 1982. A generalization of algebraic surface drawing. ACM TOG 1, 3 (1982), 235–256. Google ScholarDigital Library
8. Edward Chang, Shiufun Cheung, and Davis Y Pan. 1999. Color filter array recovery using a threshold-based variable number of gradients. In Sensors, Cameras, and Applications for Digital Photography, Vol. 3650. 36–44.Google ScholarCross Ref
9. CIPA. 2018. CIPA Report. www.cipa.jp/stats/report_e.html. {Online; accessed 29-Nov-2018}.Google Scholar
10. Sabrina Dammertz and Alexander Keller. 2008. Image synthesis by rank-1 lattices. In Monte Carlo and Quasi-Monte Carlo Methods 2006. 217–236.Google Scholar
11. Michael Drobot. 2014. Hybrid reconstruction anti-aliasing. In ACM SIGGRAPH Courses.Google Scholar
12. Joan Duran and Antoni Buades. 2014. Self-similarity and spectral correlation adaptive algorithm for color demosaicking. IEEE Trans. Image Processing 23, 9 (2014), 4031–4040.Google ScholarCross Ref
13. Michael Elad and Arie Feuer. 1997. Restoration of a single superresolution image from several blurred, noisy, and undersampled measured images. IEEE Trans. Image Processing 6, 12 (1997), 1646–1658. Google ScholarDigital Library
14. Sina Farsiu, Michael Elad, and Peyman Milanfar. 2006. Multiframe demosaicing and super-resolution of color images. IEEE Trans. Image Processing 15, 1 (2006), 141–159. Google ScholarDigital Library
15. David J Fleet and Allan D Jepson. 1990. Computation of component image velocity from local phase information. IJCV 5, 1 (1990), 77–104. Google ScholarDigital Library
16. Flickr. 2017. Top Devices of 2017 on Flickr. https://blog.flickr.net/en/2017/12/07/top-devices-of-2017/. {Online; accessed 11-Jan-2019}.Google Scholar
17. Alessandro Foi, Mejdi Trimeche, Vladimir Katkovnik, and Karen Egiazarian. 2008. Practical Poissonian-Gaussian noise modeling and fitting for single-image raw-data. IEEE Trans. Image Processing 17, 10 (2008), 1737–1754. Google ScholarDigital Library
18. Michaël Gharbi, Gaurav Chaurasia, Sylvain Paris, and Frédo Durand. 2016. Deep joint demosaicking and denoising. ACM TOG 35, 6 (2016), 191. Google ScholarDigital Library
19. Clément Godard, Kevin Matzen, and Matt Uyttendaele. 2018. Deep burst denoising. In Proc. ECCV, Vol. 11219. 560–577.Google Scholar
20. Tomomasa Gotoh and Masatoshi Okutomi. 2004. Direct super-resolution and registration using raw CFA images. In Proc. CVPR, Vol. 2. Google ScholarDigital Library
21. Chris Harris and Mike Stephens. 1988. A combined corner and edge detector.. In Alvey Vision Conference, Vol. 15. 10–5244.Google ScholarCross Ref
22. Samuel W Hasinoff, Dillon Sharlet, Ryan Geiss, Andrew Adams, Jonathan T Barron, Florian Kainz, Jiawen Chen, and Marc Levoy. 2016. Burst photography for high dynamic range and low-light imaging on mobile cameras. ACM TOG 35, 6 (2016), 192. Google ScholarDigital Library
23. Sung Hee Park and Marc Levoy. 2014. Gyro-based multi-image deconvolution for removing handshake blur. In Proc. ICCV. 3366–3373.Google ScholarDigital Library
24. Felix Heide, Markus Steinberger, Yun-Ta Tsai, Mushfiqur Rouf, Dawid Pajkk, Dikpal Reddy, Orazio Gallo, Jing Liu, Wolfgang Heidrich, et al. 2014. FlexISP: A flexible camera image processing framework. ACM TOG 33, 6 (2014), 231. Google ScholarDigital Library
25. Carl W Helstrom. 1969. Detection and resolution of incoherent objects by a background-limited optical system. JOSA 59, 2 (1969), 164–175.Google ScholarCross Ref
26. Robert Herzog, Elmar Eisemann, Karol Myszkowski, and H-P Seidel. 2010. Spatio-temporal upsampling on the GPU. In Proceedings of the 2010 ACM SIGGRAPH symposium on Interactive 3D Graphics and Games. 91–98. Google ScholarDigital Library
27. Keigo Hirakawa and Thomas W Parks. 2005. Adaptive homogeneity-directed demosaicing algorithm. IEEE Trans. Image Processing 14, 3 (2005), 360–369. Google ScholarDigital Library
28. Keigo Hirakawa and Thomas W Parks. 2006. Joint demosaicing and denoising. IEEE Trans. Image Processing 15, 8 (2006), 2146–2157. Google ScholarDigital Library
29. Terence D Hunt. 2004. Image Super-Resolution Using Adaptive 2-D Gaussian Basis Function Interpolation. Technical Report. Air Force Inst of Tech Wright-Patterson AFB OH School of Engineering.Google Scholar
30. Michal Irani and Shmuel Peleg. 1991. Improving resolution by image registration. CVGIP: Graphical models and image processing 53, 3 (1991), 231–239. Google ScholarDigital Library
31. Jorge Jimenez, Diego Gutierrez, Jason Yang, Alexander Reshetov, Pete Demoreuille, Tobias Berghoff, Cedric Perthuis, Henry Yu, Morgan McGuire, Timothy Lottes, Hugh Malan, Emil Persson, Dmitry Andreev, and Tiago Sousa. 2011. Filtering Approaches for Real-Time Anti-Aliasing. In ACM SIGGRAPH Courses. Google ScholarDigital Library
32. Takeo Kanade and Masatoshi Okutomi. 1991. A stereo matching algorithm with an adaptive window: Theory and experiment. In Proc. IEEE ICRA. IEEE, 1088–1095.Google ScholarCross Ref
33. Brian Karis. 2014. High-Quality Temporal Supersampling. In ACM SIGGRAPH Courses.Google Scholar
34. Darwin T Kuan, Alexander A Sawchuk, Timothy C Strand, and Pierre Chavel. 1985. Adaptive noise smoothing filter for images with signal-dependent noise. IEEE Trans. PAMI 2 (1985), 165–177. Google ScholarDigital Library
35. Yeon Ju Lee and Jungho Yoon. 2010. Nonlinear image upsampling method based on radial basis function interpolation. IEEE Trans. Image Processing 19, 10 (2010), 2682–2692. Google ScholarDigital Library
36. Brian Leung, Gwanggil Jeon, and Eric Dubois. 2011. Least-squares luma-chroma demultiplexing algorithm for Bayer demosaicking. IEEE Trans. Image Processing 20, 7 (2011), 1885–1894. Google ScholarDigital Library
37. Tzu-Mao Li, Michaël Gharbi, Andrew Adams, Frédo Durand, and Jonathan Ragan-Kelley. 2018. Differentiable programming for image processing and deep learning in Halide. ACM Trans. Graph. (Proc. SIGGRAPH) 37, 4 (2018), 139:1–139:13. Google ScholarDigital Library
38. Xin Li, Bahadir Gunturk, and Lei Zhang. 2008. Image demosaicing: A systematic survey. In Visual Communications and Image Processing 2008, Vol. 6822. 68221J.Google Scholar
39. Ce Liu and Deqing Sun. 2011. A Bayesian approach to adaptive video super resolution. In Proc. CVPR. IEEE, 209–216. Google ScholarDigital Library
40. Xiao-Ming Lu, Hari Krovi, Ranjith Nair, Saikat Guha, and Jeffrey H Shapiro. 2018. Quantum-optimal detection of one-versus-two incoherent optical sources with arbitrary separation. arXiv preprint arXiv:1802.02300 (2018).Google Scholar
41. Bruce D Lucas and Takeo Kanade. 1981. An iterative image registration technique with an application to stereo vision. (1981).Google Scholar
42. Rastislav Lukac and Konstantinos N Plataniotis. 2004. Normalized color-ratio modeling for CFA interpolation. IEEE Trans. Consumer Electronics 50, 2 (2004), 737–745. Google ScholarDigital Library
43. Hugh Malan. 2012. Real-Time Global Illumination and Reflections in Dust 514. In ACM SIGGRAPH Courses.Google Scholar
44. John Marshall and E Geoffrey Walsh. 1956. Physiological tremor. Journal of neurology, neurosurgery, and psychiatry 19, 4 (1956), 260.Google ScholarCross Ref
45. Ben Mildenhall, Jonathan T Barron, Jiawen Chen, Dillon Sharlet, Ren Ng, and Robert Carroll. 2018. Burst denoising with kernel prediction networks. In Proc. CVPR. 2502–2510.Google ScholarCross Ref
46. Yusuke Monno, Daisuke Kiku, Masayuki Tanaka, and Masatoshi Okutomi. 2015. Adaptive residual interpolation for color image demosaicking. In Proc. IEEE ICIP. 3861–3865.Google ScholarDigital Library
47. Matthias Müller, Barbara Solenthaler, Richard Keiser, and Markus Gross. 2005. Particle-based fluid-fluid interaction. In Proceedings of the 2005 ACM SIGGRAPH/Eurographics symposium on Computer animation. 237–244. Google ScholarDigital Library
48. NIH. 2018. Tremor Fact. www.ninds.nih.gov/Disorders/Patient-Caregiver-Education/Fact-Sheets/Tremor-Fact-Sheet. {Online; accessed 29-Nov-2018}.Google Scholar
49. Harry Nyquist. 1928. Certain topics in telegraph transmission theory. Transactions of the American Institute of Electrical Engineers 47, 2 (1928), 617–644.Google ScholarCross Ref
50. Athanasios Papoulis. 1977. Generalized sampling expansion. IEEE Trans. Circuits and Systems 24, 11 (1977), 652–654.Google ScholarCross Ref
51. Cameron N Riviere, R Scott Rader, and Nitish V Thakor. 1998. Adaptive cancelling of physiological tremor for improved precision in microsurgery. IEEE Trans. Biomedical Engineering 45, 7 (1998), 839–846.Google ScholarCross Ref
52. Dirk Robinson and Peyman Milanfar. 2004. Fundamental performance limits in image registration. IEEE Trans. Image Processing 13, 9 (2004), 1185–1199. Google ScholarDigital Library
53. Dirk Robinson and Peyman Milanfar. 2006. Statistical performance analysis of super-resolution. IEEE Trans. Image Processing 15, 6 (2006), 1413–1428. Google ScholarDigital Library
54. Yaniv Romano, John Isidoro, and Peyman Milanfar. 2017. RAISR: Rapid and accurate image super resolution. IEEE Trans. Computational Imaging 3, 1 (2017), 110–125.Google ScholarCross Ref
55. Mehdi S. M. Sajjadi, Raviteja Vemulapalli, and Matthew Brown. 2018. Frame-recurrent video super-resolution. In Proc. CVPR. 6626–6634.Google ScholarCross Ref
56. Marco Salvi. 2016. An excursion in temporal super sampling. GDC2016 From the Lab Bench: Real-Time Rendering Advances from NVIDIA Research (2016).Google Scholar
57. E. A. Schäfer. 1886. On the rhythm of muscular response to volitional impulses in man. The Journal of Physiology 7, 2 (1886), 111–117.Google ScholarCross Ref
58. Morteza Shahram and Peyman Milanfar. 2006. Statistical and information-theoretic analysis of resolution in imaging. IEEE Trans. Information Theory 52, 8 (2006), 3411–3437. Google ScholarDigital Library
59. Masao Shimiziu and Masatoshi Okutomi. 2005. Sub-pixel estimation error cancellation on area-based matching. IJCV 63 (2005), 207–224. Issue 3. Google ScholarDigital Library
60. Tiago Sousa. 2013. Graphics Gems CryENGINE3. In ACM SIGGRAPH Courses.Google Scholar
61. Molly M Sturman, David E Vaillancourt, and Daniel M Corcos. 2005. Effects of aging on the regularity of physiological tremor. Journal of Neurophysiology 93, 6 (2005), 3064–3074.Google ScholarCross Ref
62. H Takeda, S Farsiu, and P Milanfar. 2006. Robust kernel regression for restoration and reconstruction of images from sparse noisy data. Proc. IEEE ICIP (2006), 1257–1260.Google ScholarCross Ref
63. H Takeda, S Farsiu, and P Milanfar. 2007. Kernel regression for image processing and reconstruction. IEEE Trans. Image Processing 16, 2 (2007), 349. Google ScholarDigital Library
64. Hanlin Tan, Xiangrong Zeng, Shiming Lai, Yu Liu, and Maojun Zhang. 2017. Joint demosaicing and denoising of noisy bayer images with ADMM. In Proc. IEEE ICIP. 2951–2955.Google ScholarCross Ref
65. R.Y. Tsai and T.S. Huang. 1984. Multiframe image restoration and registration. Advance Computer Visual and Image Processing 1 (1984), 317–339.Google Scholar
66. Patrick Vandewalle, Karim Krichane, David Alleysson, and Sabine Süsstrunk. 2007. Joint demosaicing and super-resolution imaging from a set of unregistered aliased images. In Digital Photography III, Vol. 6502. International Society for Optics and Photonics, 65020A.Google ScholarCross Ref
67. Neal Wadhwa, Rahul Garg, David E Jacobs, Bryan E Feldman, Nori Kanazawa, Robert Carroll, Yair Movshovitz-Attias, Jonathan T Barron, Yael Pritch, and Marc Levoy. 2018. Synthetic depth-of-field with a single-camera mobile phone. ACM TOG 37, 4 (2018), 64. Google ScholarDigital Library
68. Hermann Weyl. 1910. Über die Gibbs’sche Erscheinung und verwandte Konvergenzphänomene. Rendiconti del Circolo Matematico di Palermo (1884–1940) 30, 1 (01 Dec 1910), 377–407.Google Scholar
69. Bartlomiej Wronski and Peyman Milanfar. 2018. See Better and Further with Super Res Zoom on the Pixel 3. https://ai.googleblog.com/2018/10/see-better-and-further-with-super-res.html.Google Scholar
70. Xiaolin Wu and Lei Zhang. 2006. Temporal color video demosaicking via motion estimation and data fusion. IEEE Trans. Circuits and Systems for Video Technology 16 (2006). Issue 2. Google ScholarDigital Library
71. J Yen. 1956. On nonuniform sampling of bandwidth-limited signals. IRE Trans. Circuit Theory 3, 4 (1956), 251–257.Google ScholarCross Ref
72. Jihun Yu and Greg Turk. 2013. Reconstructing surfaces of particle-based fluids using anisotropic kernels. ACM TOG 32, 1 (2013), 5. Google ScholarDigital Library
73. Lei Zhang, Xiaolin Wu, Antoni Buades, and Xin Li. 2011. Color demosaicking bylocal directional interpolation and nonlocal adaptive thresholding. Journal of Electronic imaging 20, 2 (2011), 023016.Google ScholarCross Ref
