“High-Quality Image Deblurring With Panchromatic Pixels” by Wang, Hou, Border, Qin and Miller

Image deblurring has been a very challenging problem in recent decades. In this article, we propose a high-quality image deblurring method with a novel image prior based on a new imaging system. The imaging system has a newly designed sensor pattern achieved by adding panchromatic (pan) pixels to the conventional Bayer pattern. Since these pan pixels are sensitive to all wavelengths of visible light, they collect a significantly higher proportion of the light striking the sensor. A new demosaicing algorithm is also proposed to restore full-resolution images from pixels on the sensor. The shutter speed of pan pixels is controllable to users. Therefore, we can produce multiple images with different exposures. When long exposure is needed under dim light, we read pan pixels twice in one shot: one with short exposure and the other with long exposure. The long-exposure image is often blurred, while the short-exposure image can be sharp and noisy. The short-exposure image plays an important role in deblurring, since it is sharp and there is no alignment problem for the one-shot image pair. For the algorithmic aspect, our method runs in a two-step maximum-a-posteriori (MAP) fashion under a joint minimization of the blur kernel and the deblurred image. The algorithm exploits a combined image prior with a statistical part and a spatial part, which is powerful in ringing controls. Extensive experiments under various conditions and settings are conducted to demonstrate the performance of our method.

Agrawal, A., Xu, Y., and Raskar, R. 2009. Invertible motion blur in video. ACM Trans. Graph. 28, 3, 95:1–95:8. Google ScholarDigital Library
Alleysson, D., Süsstrunk, S., and Hérault, J. 2005. Linear demosaicing inspired by the human visual system. IEEE Trans. Image Process. 14, 4, 439–449. Google ScholarDigital Library
Bayer, B. 1976. Color imaging array. US Patent 3,971,065.Google Scholar
Ben-Ezra, M. and Nayar, S. 2003. Motion deblurring using hybrid imaging. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 657–664. Google ScholarDigital Library
Border, J., Anderson, T., and Deever, A. 2009. Determining and correcting for imaging device motion during an exposure. US Patent application US2009/0021588A1.Google Scholar
Cai, J., Ji, H., Liu, C., and Shen, Z. 2009a. Blind motion deblurring from a single image using sparse approximation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 104–111.Google Scholar
Cai, J., Ji, H., Liu, C., and Shen, Z. 2009b. High-quality curvelet-based motion deblurring from an image pair. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1566–1573.Google Scholar
Chen, J., Yuan, L., Tang, C., and Quan, L. 2008. Robust dual motion deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1–8.Google Scholar
Cho, S. and Lee, S. 2009. Fast motion deblurring. ACM Trans. Graph. 28, 5, 145:1–145:8. Google ScholarDigital Library
Compton, J. and Hamilton, J. 2007. Capturing images under varying lighting conditions. US Patent application 2007/0046807A1.Google Scholar
Fergus, R., Singh, B., Hertzmann, A., Roweis, S., and Freeman, W. 2006. Removing camera shake from a single photograph. ACM Trans. Graph. 25, 3, 787–794. Google ScholarDigital Library
Hou, T., Wang, S., and Qin, H. 2011. Image deconvolution with multi-stage convex relaxation and its perceptual evaluation. IEEE Trans. Image Process. 20, 12, 3383–3392. Google ScholarDigital Library
Joshi, N., Kang, S., Zitnick, C., and Szeliski, R. 2010. Image deblurring using inertial measurement sensors. ACM Trans. Graph. 29, 4, 30:1–30:9. Google ScholarDigital Library
Joshi, N., Zitnick, C., Szeliski, R., and Kriegman, D. 2009. Image deblurring and denoising using color priors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1550–1557.Google Scholar
Krishnan, D. and Fergus, R. 2009. Fast image deconvolution using hyper-laplacian priors. In Proceedings of the Neural Information Processing Systems Conference.Google Scholar
Levin, A., Fergus, R., Durand, F., and Freeman, W. 2007. Image and depth from a conventional camera with a coded aperture. ACM Trans. Graph. 26, 6, 70–77. Google ScholarDigital Library
Levin, A., Sand, P., Cho, T., Durand, F., and Freeman, W. 2008. Motion-invariant photograph. ACM Trans. Graph. 27, 3, 71:1–71:9. Google ScholarDigital Library
Levin, A., Weiss, Y., Durand, F., and Freeman, W. 2009. Understanding and evaluating blind deconvolution algorithms. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1964–1971.Google Scholar
Lim, S. and Silverstein, D. 2006. Method for deblurring an image. US Patent application 2006/0187308A1.Google Scholar
Lucy, L. 1974. Bayesian-Based iterative method of image restoration. J. Astron. 79, 745–754.Google ScholarCross Ref
Muramatsu, A. 1989. Color video signal generating device using monochrome and color image sensors having different resolutions to form a luminance signal. US Patent 4,876,591.Google Scholar
Nayar, S. and Narasimhan, S. 2002. Assorted pixels: Multi-Sampled imaging with structural models. In Proceedings of the European Conference on Computer Vision. 472–479. Google ScholarDigital Library
Raskar, R., Agrawal, A., and Tumblin, J. 2006. Coded exposure photography:motion deblurring using fluttered shutter. ACM Trans. Graph. 25, 3, 795–804. Google ScholarDigital Library
Shan, Q., Jia, J., and Agarwala, A. 2008. High-quality motion deblurring from a single image. ACM Trans. Graph. 27, 3, 73:1–73:10. Google ScholarDigital Library
Susanu, G., Petrescu, S., Nanu, F., Capata, A., and Corcoran, P. 2009. Rgbw sensor array. US Patent application 2009/0167893A1.Google Scholar
Tomasi, C. and Manduchi, R. 1998. Bilateral filtering for gray and color images. In Proceedings of the International Conference on Computer Vision. 839–846. Google ScholarDigital Library
Wang, S., Hou, T., and Miller, R. 2011. Image deblurring using panchromatic pixels. US Patent application 2011/0090378A1.Google Scholar
Whyte, O., Sivc, J., Zisserman, A., and Ponce, J. 2010. Non-uniform deblurring for shaken images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 491–498.Google Scholar
Xu, L. and Jia, J. 2010. Two-phase kernel estimation for robust motion deblurring. In Proceedings of the European Conference on Computer Vision. 157–170. Google ScholarDigital Library
Yuan, L., Sun, J., Quan, L., and Shum, H. 2007. Image deblurring with blurred/noisy image pairs. ACM Trans. Graph. 26, 3. Google ScholarDigital Library
Yuan, L., Sun, J., Quan, L., and Shum, H. 2008. Progressive inter-scale and intra-scale non-blind image deconvolution. ACM Trans. Graph. 27, 3, 74:1–74:10. Google ScholarDigital Library
Zhou, C. and Nayar, S. 2009. What are good apertures for defocus deblurring? In Proceedings of the International Conference on Computational Photography. 1–8.Google Scholar