“FlexISP: a flexible camera image processing framework” by Heide, Steinberger, Tsai, Rouf, Pająk, et al. …
Conference:
Type(s):
Title:
- FlexISP: a flexible camera image processing framework
Session/Category Title: Digital Photography
Presenter(s)/Author(s):
Abstract:
Conventional pipelines for capturing, displaying, and storing images are usually defined as a series of cascaded modules, each responsible for addressing a particular problem. While this divide-and-conquer approach offers many benefits, it also introduces a cumulative error, as each step in the pipeline only considers the output of the previous step, not the original sensor data. We propose an end-to-end system that is aware of the camera and image model, enforces natural-image priors, while jointly accounting for common image processing steps like demosaicking, denoising, deconvolution, and so forth, all directly in a given output representation (e.g., YUV, DCT). Our system is flexible and we demonstrate it on regular Bayer images as well as images from custom sensors. In all cases, we achieve large improvements in image quality and signal reconstruction compared to state-of-the-art techniques. Finally, we show that our approach is capable of very efficiently handling high-resolution images, making even mobile implementations feasible.
References:
1. Adams, A., Talvala, E.-V., Park, S. H., Jacobs, D., Ajdin, B., Gelfand, N., Dolson, J., Vaquero, D., Baek, J., Tico, M., Lensch, H. P. A., Matusik, W., Pulli, K., Horowitz, M., and Levoy, M. 2010. The Frankencamera: an experimental platform for computational photography. ACM TOG 29, 4.
2. Afonso, M. V., Bioucas-Dias, J. M., and Figueiredo, M. A. 2010. Fast image recovery using variable splitting and constrained optimization. IEEE TIP 19, 9.
3. Ajdin, B., Hullin, M. B., Fuchs, C., Seidel, H.-P., and Lensch, H. 2008. Demosaicing by smoothing along 1d features. In CVPR.
4. Bennett, E. P., Uyttendaele, M., Zitnick, C., Szeliski, R., and Kang, S. 2006. Video and image bayesian demosaicing with a two color image prior. In ECCV.
5. Brauers, J., Seiler, C., and Aach, T. 2010. Direct psf estimation using a random noise target. In Electronic Imaging.
6. Buades, A., Coll, B., and Morel, J. M. 2005. A non-local algorithm for image denoising. In CVPR.
7. Buades, T., Lou, Y., Morel, J. M., and Tang, Z. 2009. A note on multi-image denoising. In Int. Workshop on Local and Non-Local Approximation in Image Processing.
8. Buades, A., Coll, B., and Morel, J.-M. 2011. Non-Local Means Denoising. Image Processing On Line.
9. Candès, E., and Donoho, D. 1999. Curvelets: A surprisingly effective nonadaptive representation of objects with edges. In Curves and Surfaces. Vanderbilt University Press.
10. Chambolle, A., and Pock, T. 2011. A first-order primal-dual algorithm for convex problems with applications to imaging. Journal of Mathematical Imaging and Vision 40, 1.
11. Chatterjee, P., Joshi, N., Kang, S. B., and Matsushita, Y. 2011. Noise suppression in low-light images through joint denoising and demosaicing. In CVPR.
12. Coifman, R. R., and Donoho, D. L. 1995. Translation-invariant denoising. Tech. Rep. 475, Stanford University.
13. Dabov, K., Foi, A., Katkovnik, V., and Egiazarian, K. 2007. Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE TIP 16, 8.
14. Dabov, K., Foi, A., and Egiazarian, K. 2007. Video de-noising by sparse 3d transform-domain collaborative filtering. In European Signal Processing Conference.
15. Dabov, K., Foi, A., Katkovnik, V., and Egiazarian, K. 2008. Image restoration by sparse 3d transform-domain collaborative filtering. In Electronic Imaging.
16. Danielyan, A., Katkovnik, V., and Egiazarian, K. 2012. BM3D frames and variational image deblurring. IEEE TIP 21, 4.
17. Egiazarian, K. O., Astola, J., Helsingius, M., and Kuosmanen, P. 1999. Adaptive denoising and lossy compression of images in transform domain. J. Electronic Imaging 8, 3.Cross Ref
18. Farsiu, S., Elad, M., and Milanfar, P. 2006. Multiframe demosaicing and superresolution of color image. IEEE TIP 15, 1.
19. Foi, A., Katkovnik, V., and Egiazarian, K. 2007. Pointwise shape-adaptive DCT for high-quality denoising and deblocking of grayscale and color images. IEEE TIP 16, 5.
20. Gu, J., Hitomi, Y., Mitsunaga, T., and Nayar, S. 2010. Coded rolling shutter photography: Flexible space-time sampling. In ICCP.
21. Hajisharif, S., Kronander, J., and Unger, J. 2014. HDR reconstruction for alternating gain (ISO) sensor readout. In Proc. of Eurographics (short paper).
22. Heide, F., Rouf, M., Hullin, M. B., Labitzke, B., Heidrich, W., and Kolb, A. 2013. High-quality computational imaging through simple lenses. ACM TOG 32, 5.
23. Irani, M., and Peleg, S. 1991. Improving resolution by image registration. Graphical Models and Image Processing 53, 3.
24. Jeon, G., and Dubois, E. 2013. Demosaicking of noisy Bayer-sampled color images with least-squares luma-chroma demultiplexing and noise level estimation. IEEE TIP 22, 1.
25. Joulin, A., and Kang, S. B. 2013. Recovering stereo pairs from anaglyphs. In CVPR.
26. Krishnan, D., and Fergus, R. 2009. Fast image deconvolution using hyper-laplacian priors. In NIPS.
27. Levin, A., Fergus, R., Durand, F., and Freeman, W. T. 2007. Deconvolution using natural image priors. ACM Transactions on Graphics (TOG) 26, 3.
28. Liu, C., Yuen, J., Torralba, A., Sivic, J., and Freeman, W. T. 2008. Sift flow: Dense correspondence across different scenes. In ECCV.
29. Liu, C. 2009. Beyond Pixels: Exploring New Representations and Applications for Motion Analysis. PhD thesis, MIT.
30. Mitra, K., Cossairt, O., and Veeraraghavan, A. 2014. A framework for analysis of computational imaging systems: Role of signal prior, sensor noise and multiplexing. ieee PAMI 36, 10.
31. Mosseri, I., Zontak, M., and Irani, M. 2013. Combining the power of internal and external denoising. In ICCP.
32. Narasimhan, S. G., and Nayar, S. K. 2005. Enhancing resolution along multiple imaging dimensions using assorted pixels. IEEE PAMI 27, 4.
33. Nayar, S., and Branzoi, V. 2003. Adaptive dynamic range imaging: Optical control of pixel exposures over space and time. In ICCV.
34. Oymak, S., and Hassibi, B. 2013. Sharp MSE bounds for proximal denoising. arXiv preprint arXiv:1305.2714.
35. Parikh, N., and Boyd, S. 2013. Proximal algorithms. Foundations and Trends in Optimization 1, 3.
36. Ramanath, R., Snyder, W. E., Yoo, Y., and Drew, M. S. 2005. Color image processing pipeline in digital still cameras. IEEE Signal Processing Magazine 22, 1.Cross Ref
37. Reinhard, E., Ward, G., Pattanaik, S., Debevec, P., Heidrich, W., and Myszkowski, K. 2010. High dynamic range imaging: acquisition, display, and image-based lighting. Morgan Kaufmann.
38. Roth, S., and Black, M. J. 2009. Fields of experts. International Journal of Computer Vision 82, 2.
39. Rudin, L. I., Osher, S., and Fatemi, E. 1992. Nonlinear total variation based noise removal algorithms. Physica D: Nonlinear Phenomena 60, 1.
40. Schuler, C. J., Hirsch, M., Harmeling, S., and Schölkopf, B. 2011. Non-stationary correction of optical aberrations. In ICCV.
41. Schuler, C. J., Burger, H. C., Harmeling, S., and Schölkopf, B. 2013. A machine learning approach for non-blind image deconvolution. In CVPR.
42. Shao, L., Yan, R., Li, X., and Liu, Y. 2013. From heuristic optimization to dictionary learning: A review and comprehensive comparison of image denoising algorithms. IEEE Transactions on Cybernetics, 99.
43. Starck, J.-L., Murtagh, F., and Bijaoui, A. 1998. Image Processing and Data Analysis: The Multiscale Approach. Cambridge University Press, Cambridge.
44. Tico, M., and Pulli, K. 2009. Image enhancement method via blur and noisy image fusion. In ICIP.
45. Tico, M. 2008. Multiframe image denoising and stabilization. In European Signal Processing Conference.
46. Tsai, Y.-T., Steinberger, M., Pająk, D., and Pulli, K. 2014. Fast ANN for high-quality collaborative filtering. In High-Performance Graphics.
47. Venkatakrishnan, S. V., Bouman, C. A., and Wohlberg, B. 2013. Plug-and-play priors for model based reconstruction. In IEEE GlobalSIP.
48. Venkataraman, K., Lelescu, D., Duparré, J., McMahon, A., Molina, G., Chatterjee, P., Mullis, R., and Nayar, S. 2013. Picam: an ultra-thin high performance monolithic camera array. ACM TOG 32, 6.
49. Wallace, G. K. 1991. The JPEG still picture compression standard. Communications of the ACM 34, 4.
50. Wilburn, B., Joshi, N., Vaish, V., Talvala, E.-V., Antunez, E., Barth, A., Adams, A., Horowitz, M., and Levoy, M. 2005. High performance imaging using large camera arrays. ACM TOG 24, 3.
51. Wu, X., and Zhang, D. 2006. Improvement of color video demosaicking in temporal domain. IEEE TIP 15, 10.
52. Xu, L., and Jia, J. 2010. Two-phase kernel estimation for robust motion deblurring. In ECCV.
53. Zhang, L., Wu, X., Buades, A., and Li, X. 2011. Color demosaicking by local directional interpolation and nonlocal adaptive thresholding. Journal of Electronic Imaging 20, 2.
54. Zoran, D., and Weiss, Y. 2011. From learning models of natural image patches to whole image restoration. In ICCV.
