“Face deblurring using dual camera fusion on mobile phones” by Lai, Shih, Chu, Wu, Tsai, et al. …

Motion blur of fast-moving subjects is a longstanding problem in photography and very common on mobile phones due to limited light collection efficiency, particularly in low-light conditions. While we have witnessed great progress in image deblurring in recent years, most methods require significant computational power and have limitations in processing high-resolution photos with severe local motions. To this end, we develop a novel face deblurring system based on the dual camera fusion technique for mobile phones. The system detects subject motion to dynamically enable a reference camera, e.g., ultrawide angle camera commonly available on recent premium phones, and captures an auxiliary photo with faster shutter settings. While the main shot is low noise but blurry (Figure 1(a)), the reference shot is sharp but noisy (Figure 1(b)). We learn ML models to align and fuse these two shots and output a clear photo without motion blur (Figure 1(c)). Our algorithm runs efficiently on Google Pixel 6, which takes 463 ms overhead per shot. Our experiments demonstrate the advantage and robustness of our system against alternative single-image, multi-frame, face-specific, and video deblurring algorithms as well as commercial products. To the best of our knowledge, our work is the first mobile solution for face motion deblurring that works reliably and robustly over thousands of images in diverse motion and lighting conditions.

1. 2021. Android Camera2 API. https://developer.android.com/guide/topics/media/camera.Google Scholar
2. Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. TensorFlow: A System for Large-Scale Machine Learning. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16). 265–283.Google ScholarDigital Library
3. Miika Aittala and Frédo Durand. 2018. Burst image deblurring using permutation invariant convolutional neural networks. In ECCV. 731–747.Google Scholar
4. Giacomo Boracchi and Alessandro Foi. 2012. Modeling the performance of image restoration from motion blur. IEEE TIP 21, 8 (2012), 3502–3517.Google Scholar
5. Daniel J Butler, Jonas Wulff, Garrett B Stanley, and Michael J Black. 2012. A naturalistic open source movie for optical flow evaluation. In ECCV. 611–625.Google Scholar
6. Meng Chang, Huajun Feng, Zhihai Xu, and Qi Li. 2021. Low-light image restoration with short-and long-exposure raw pairs. IEEE Transactions on Multimedia.Google Scholar
7. Qifeng Chen and Vladlen Koltun. 2017. Photographic image synthesis with cascaded refinement networks. In ICCV. 1511–1520.Google Scholar
8. Sunghyun Cho and Seungyong Lee. 2009. Fast motion deblurring. In ACM TOG. 1–8.Google Scholar
9. Sunghyun Cho, Jue Wang, and Seungyong Lee. 2012. Video deblurring for hand-held cameras using patch-based synthesis. ACM SIGGRAPH 31, 4.Google ScholarDigital Library
10. Sung-Jin Cho, Seo-Won Ji, Jun-Pyo Hong, Seung-Won Jung, and Sung-Jea Ko. 2021. Rethinking coarse-to-fine approach in single image deblurring. In ICCV. 4641–4650.Google Scholar
11. Mauricio Delbracio, Ignacio Garcia-Dorado, Sungjoon Choi, Damien Kelly, and Peyman Milanfar. 2021. Polyblur: Removing mild blur by polynomial reblurring. IEEE Transactions on Computational Imaging 7, 837–848.Google ScholarCross Ref
12. Mauricio Delbracio and Guillermo Sapiro. 2015a. Burst deblurring: Removing camera shake through Fourier burst accumulation. In CVPR. 2385–2393.Google Scholar
13. Mauricio Delbracio and Guillermo Sapiro. 2015b. Hand-held video deblurring via efficient fourier aggregation. Transactions on Computational Imaging 1, 4.Google Scholar
14. Senyou Deng, Wenqi Ren, Yanyang Yan, Tao Wang, Fenglong Song, and Xiaochun Cao. 2021. Multi-Scale Separable Network for Ultra-High-Definition Video Deblurring. In CVPR. 14030–14039.Google Scholar
15. Henry Dietz and Paul Eberhart. 2019. Shuttering methods and the artifacts they produce. Electronic Imaging 2019, 4.Google Scholar
16. Jiangxin Dong, Jinshan Pan, Zhixun Su, and Ming-Hsuan Yang. 2017. Blind image deblurring with outlier handling. In ICCV. 2478–2486.Google Scholar
17. Jana Ehmann, Lun-Cheng Chu, Sung-Fang Tsai, and Chia-Kai Liang. 2018. Real-time video denoising on mobile phones. In ICIP. 505–509.Google Scholar
18. Rob Fergus, Barun Singh, Aaron Hertzmann, Sam T Roweis, and William T Freeman. 2006. Removing camera shake from a single photograph. In ACM SIGGRAPH. 787–794.Google Scholar
19. Alessandro Foi, Mejdi Trimeche, Vladimir Katkovnik, and Karen Egiazarian. 2008.Google Scholar
20. Practical Poissonian-Gaussian noise modeling and fitting for single-image raw-data. IEEE TIP 17, 10 (2008), 1737–1754.Google Scholar
21. Hongyun Gao, Xin Tao, Xiaoyong Shen, and Jiaya Jia. 2019. Dynamic scene deblurring with parameter selective sharing and nested skip connections. In CVPR. 3848–3856.Google Scholar
22. Amnon Geifman. 2020. The Correct Way to Measure Inference Time of Deep Neural Networks. https://deci.ai/resources/blog/measure-inference-time-deep-neural-networks/.Google Scholar
23. Dong Gong, Jie Yang, Lingqiao Liu, Yanning Zhang, Ian Reid, Chunhua Shen, Anton Van Den Hengel, and Qinfeng Shi. 2017. From motion blur to motion flow: A deep learning solution for removing heterogeneous motion blur. In CVPR. 2319–2328.Google Scholar
24. Chunzhi Gu, Xuequan Lu, Ying He, and Chao Zhang. 2020. Blur removal via blurrednoisy image pair. IEEE TIP 30, 345–359.Google Scholar
25. Yoav Hacohen, Eli Shechtman, and Dani Lischinski. 2013. Deblurring by example using dense correspondence. In ICCV. 2384–2391.Google Scholar
26. Samuel W Hasinoff, Dillon Sharlet, Ryan Geiss, Andrew Adams, Jonathan T Barron, Florian Kainz, Jiawen Chen, and Marc Levoy. 2016. Burst photography for high dynamic range and low-light imaging on mobile cameras. ACM SIGGRAPH 35, 6.Google ScholarDigital Library
27. Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. 2017. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv:1704.04861.Google Scholar
28. Xiaobin Hu, Wenqi Ren, Kaicheng Yu, Kaihao Zhang, Xiaochun Cao, Wei Liu, and Bjoern Menze. 2021. Pyramid architecture search for real-time image deblurring. In ICCV. 4298–4307.Google Scholar
29. Zhe Hu, Sunghyun Cho, Jue Wang, and Ming-Hsuan Yang. 2014. Deblurring low-light images with light streaks. In ICCV. 3382–3389.Google Scholar
30. Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. 2017. Densely connected convolutional networks. In CVPR. 4700–4708.Google Scholar
31. Tae Hyun Kim and Kyoung Mu Lee. 2015. Generalized video deblurring for dynamic scenes. In CVPR. 5426–5434.Google Scholar
32. Tae Hyun Kim, Kyoung Mu Lee, Bernhard Scholkopf, and Michael Hirsch. 2017. Online video deblurring via dynamic temporal blending network. In ICCV. 4038–4047.Google Scholar
33. Adam Kaufman and Raanan Fattal. 2020. Deblurring using analysis-synthesis networks pair. In CVPR. 5811–5820.Google Scholar
34. Tae Hyun Kim, Mehdi SM Sajjadi, Michael Hirsch, and Bernhard Scholkopf. 2018. Spatio-temporal transformer network for video restoration. In ECCV. 106–122.Google Scholar
35. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv:1412.6980.Google Scholar
36. Jaihyun Koh, Jangho Lee, and Sungroh Yoon. 2021. Single-image deblurring with neural networks: A comparative survey. CVIU 203.Google Scholar
37. Dilip Krishnan, Terence Tay, and Rob Fergus. 2011. Blind deconvolution using a normalized sparsity measure. In CVPR. 233–240.Google Scholar
38. Orest Kupyn, Volodymyr Budzan, Mykola Mykhailych, Dmytro Mishkin, and Jiří Matas. 2018. Deblurgan: Blind motion deblurring using conditional adversarial networks. In CVPR. 8183–8192.Google Scholar
39. Orest Kupyn, Tetiana Martyniuk, Junru Wu, and Zhangyang Wang. 2019. DeblurGANv2: Deblurring (orders-of-magnitude) faster and better. In ICCV. 8878–8887.Google Scholar
40. Wei-Sheng Lai, Jian-Jiun Ding, Yen-Yu Lin, and Yung-Yu Chuang. 2015. Blur kernel estimation using normalized color-line prior. In ICCV. 64–72.Google Scholar
41. Lerenhan Li, Jinshan Pan, Wei-Sheng Lai, Changxin Gao, Nong Sang, and Ming-Hsuan Yang. 2018. Learning a discriminative prior for blind image deblurring. In CVPR. 6616–6625.Google Scholar
42. Lerenhan Li, Jinshan Pan, Wei-Sheng Lai, Changxin Gao, Nong Sang, and Ming-Hsuan Yang. 2020. Dynamic scene deblurring by depth guided model. IEEE TIP 29, 5273–5288.Google Scholar
43. Orly Liba, Kiran Murthy, Yun-Ta Tsai, Tim Brooks, Tianfan Xue, Nikhil Karnad, Qiurui He, Jonathan T Barron, Dillon Sharlet, Ryan Geiss, et al. 2019. Handheld mobile photography in very low light. ACM SIGGRAPH 38, 6.Google ScholarDigital Library
44. Songnan Lin, Jiawei Zhang, Jinshan Pan, Yicun Liu, Yongtian Wang, Jing Chen, and Jimmy Ren. 2020. Learning to deblur face images via sketch synthesis. In AAAI, Vol. 34.Google ScholarCross Ref
45. Nikolaus Mayer, Eddy Ilg, Philip Hausser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox. 2016. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In CVPR. 4040–4048.Google Scholar
46. Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. 2017. Deep multi-scale convolutional neural network for dynamic scene deblurring. In CVPR. 3883–3891.Google Scholar
47. Seungjun Nah, Sanghyun Son, and Kyoung Mu Lee. 2019. Recurrent neural networks with intra-frame iterations for video deblurring. In CVPR. 8102–8111.Google Scholar
48. Jinshan Pan, Zhe Hu, Zhixun Su, and Ming-Hsuan Yang. 2014. Deblurring face images with exemplars. In ECCV. 47–62.Google Scholar
49. Jinshan Pan, Zhouchen Lin, Zhixun Su, and Ming-Hsuan Yang. 2016a. Robust kernel estimation with outliers handling for image deblurring. In ICCV. 2800–2808.Google Scholar
50. Jinshan Pan, Deqing Sun, Hanspeter Pfister, and Ming-Hsuan Yang. 2016b. Blind image deblurring using dark channel prior. In CVPR. 1628–1636.Google Scholar
51. Tobias Plotz and Stefan Roth. 2017. Benchmarking denoising algorithms with real photographs. In CVPR. 1586–1595.Google Scholar
52. Erik Reinhard, Michael Adhikhmin, Bruce Gooch, and Peter Shirley. 2001. Color transfer between images. IEEE Computer graphics and applications 21, 5 (2001), 34–41.Google ScholarDigital Library
53. Wenqi Ren, Xiaochun Cao, Jinshan Pan, Xiaojie Guo, Wangmeng Zuo, and Ming-Hsuan Yang. 2016. Image deblurring via enhanced low-rank prior. IEEE TIP 25, 7.Google Scholar
54. Wenqi Ren, Jinshan Pan, Xiaochun Cao, and Ming-Hsuan Yang. 2017. Video deblurring via semantic segmentation and pixel-wise non-linear kernel. In ICCV. 1077–1085.Google Scholar
55. Wenqi Ren, Jiaolong Yang, Senyou Deng, David Wipf, Xiaochun Cao, and Xin Tong. 2019. Face video deblurring using 3d facial priors. In CVPR. 9388–9397.Google Scholar
56. Jaesung Rim, Haeyun Lee, Jucheol Won, and Sunghyun Cho. 2020. Real-world blur dataset for learning and benchmarking deblurring algorithms. In ECCV. 184–201.Google Scholar
57. Qi Shan, Jiaya Jia, and Aseem Agarwala. 2008. High-quality motion deblurring from a single image. ACM SIGGRAPH 27, 3.Google ScholarDigital Library
58. Ziyi Shen, Wei-Sheng Lai, Tingfa Xu, Jan Kautz, and Ming-Hsuan Yang. 2018. Deep semantic face deblurring. In CVPR. 8260–8269.Google Scholar
59. Ziyi Shen, Wei-Sheng Lai, Tingfa Xu, Jan Kautz, and Ming-Hsuan Yang. 2020. Exploiting semantics for face image deblurring. IJCV 128, 7.Google ScholarCross Ref
60. Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556.Google Scholar
61. Hyeongseok Son, Junyong Lee, Jonghyeop Lee, Sunghyun Cho, and Seungyong Lee. 2021. Recurrent video deblurring with blur-invariant motion estimation and pixel volumes. ACM SIGGRAPH 40, 5.Google ScholarDigital Library
62. Shuochen Su, Mauricio Delbracio, Jue Wang, Guillermo Sapiro, Wolfgang Heidrich, and Oliver Wang. 2017. Deep video deblurring for hand-held cameras. In CVPR. 1279–1288.Google Scholar
63. Maitreya Suin, Kuldeep Purohit, and AN Rajagopalan. 2020. Spatially-attentive patchhierarchical network for adaptive motion deblurring. In CVPR. 3606–3615.Google Scholar
64. Deqing Sun, Daniel Vlasic, Charles Herrmann, Varun Jampani, Michael Krainin, Huiwen Chang, Ramin Zabih, William T Freeman, and Ce Liu. 2021. AutoFlow: Learning a Better Training Set for Optical Flow. In CVPR. 10093–10102.Google Scholar
65. Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz. 2018. PWC-net: CNNs for optical flow using pyramid, warping, and cost volume. In CVPR. 8934–8943.Google Scholar
66. Jian Sun, Wenfei Cao, Zongben Xu, and Jean Ponce. 2015. Learning a convolutional neural network for non-uniform motion blur removal. In ICCV. 769–777.Google Scholar
67. Libin Sun, Sunghyun Cho, Jue Wang, and James Hays. 2013. Edge-based blur kernel estimation using patch priors. In ICCP. 1–8.Google Scholar
68. Hossein Talebi and Peyman Milanfar. 2018. NIMA: Neural image assessment. IEEE TIP 27, 8.Google Scholar
69. Xin Tao, Hongyun Gao, Xiaoyong Shen, Jue Wang, and Jiaya Jia. 2018. Scale-recurrent network for deep image deblurring. In CVPR. 8174–8182.Google Scholar
70. Zachary Teed and Jia Deng. 2020. Raft: Recurrent all-pairs field transforms for optical flow. In ECCV. 402–419.Google Scholar
71. Neal Wadhwa, Rahul Garg, David E Jacobs, Bryan E Feldman, Nori Kanazawa, Robert Carroll, Yair Movshovitz-Attias, Jonathan T Barron, Yael Pritch, and Marc Levoy. 2018. Synthetic depth-of-field with a single-camera mobile phone. ACM SIGGRAPH 37, 4.Google ScholarDigital Library
72. Xintao Wang, Kelvin CK Chan, Ke Yu, Chao Dong, and Chen Change Loy. 2019. EDVR: Video restoration with enhanced deformable convolutional networks. In CVPR.Google Scholar
73. Oliver Whyte, Josef Sivic, and Andrew Zisserman. 2014. Deblurring shaken and partially saturated images. IJCV 110, 2.Google ScholarDigital Library
74. Li Xu and Jiaya Jia. 2010. Two-phase kernel estimation for robust motion deblurring. In ECCV. 157–170.Google Scholar
75. Li Xu, Shicheng Zheng, and Jiaya Jia. 2013. Unnatural L0 sparse representation for natural image deblurring. In CVPR. 1107–1114.Google Scholar
76. Xiangyu Xu, Deqing Sun, Jinshan Pan, Yujin Zhang, Hanspeter Pfister, and Ming-Hsuan Yang. 2017. Learning to super-resolve blurry face and text images. In ICCV. 251–260.Google Scholar
77. Liuge Yang and Hui Ji. 2019. A variational EM framework with adaptive edge selection for blind motion deblurring. In CVPR. 10167–10176.Google Scholar
78. Rajeev Yasarla, Federico Perazzi, and Vishal M Patel. 2020. Deblurring face images using uncertainty guided multi-stream semantic networks. IEEE TIP 29 (2020), 6251–6263.Google Scholar
79. Amir Yazdanbakhsh, Kiran Seshadri, Berkin Akin, James Laudon, and Ravi Narayanaswami. 2021. An evaluation of edge TPU accelerators for convolutional neural networks. arXiv:2102.10423.Google Scholar
80. Lu Yuan, Jian Sun, Long Quan, and Heung-Yeung Shum. 2007. Image deblurring with blurred/noisy image pairs. In ACM SIGGRAPH.Google Scholar
81. Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao. 2021. Multi-stage progressive image restoration. In CVPR. 14821–14831.Google Scholar
82. Hongguang Zhang, Yuchao Dai, Hongdong Li, and Piotr Koniusz. 2019. Deep stacked hierarchical multi-patch network for image deblurring. In CVPR. 5978–5986.Google Scholar
83. Kaihao Zhang, Wenhan Luo, Yiran Zhong, Lin Ma, Bjorn Stenger, Wei Liu, and Hongdong Li. 2020. Deblurring by realistic blurring. In CVPR. 2737–2746.Google Scholar
84. Shangchen Zhou, Jiawei Zhang, Jinshan Pan, Haozhe Xie, Wangmeng Zuo, and Jimmy Ren. 2019. Spatio-temporal filter adaptive network for video deblurring. In CVPR. 2482–2491.Google Scholar
85. Shaojie Zhuo, Dong Guo, and Terence Sim. 2010. Robust flash deblurring. In CVPR. 2440–2447.Google Scholar