“GradientShop: A Gradient-Domain Optimization Framework for Image and Video Filtering” by Bhat, Zitnick, Cohen and Curless

We present an optimization framework for exploring gradient-domain solutions for image and video processing. The proposed framework unifies many of the key ideas in the gradient-domain literature under a single optimization formulation. Our hope is that this generalized framework will allow the reader to quickly gain a general understanding of the field and contribute new ideas of their own.

We propose a novel metric for measuring local gradient saliency that identifies salient gradients that give rise to long, coherent edges, even when the individual gradients are faint. We present a general weighting scheme for gradient constraints that improves the visual appearance of results. We also provide a solution for applying gradient-domain filters to videos and video streams in a coherent manner.

Finally, we demonstrate the utility of our formulation in creating effective yet simple to implement solutions for various image-processing tasks. To exercise our formulation we have created a new saliency-based sharpen filter and a pseudo image-relighting application. We also revisit and improve upon previously defined filters such as nonphotorealistic rendering, image deblocking, and sparse data interpolation over images (e.g., colorization using optimization).

1. Agrawal, A. and Raskar, R., 2007. Gradient domain manipulation techniques vision and graphics. ICCV 2007 Courses.
2. Agrawal, A., Raskar, R., Nayar, S., and Li, Y., 2005. Removing photography artifacts using gradient projection and flash exposure sampling. ACM Trans. Graph. 
3. Agrawal, A., Raskar, R., and Chellappa, R. 2006. Edge suppression by gradient field transformation using cross-projection tensors. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR’06). 2301–2308. 
4. Agrawal, A., Raskar, R., and Chellappa, R. 2006. What is the range of surface reconstructions from a gradient field. In Proceedings of the European Conference on Computer Vision (ECCV). Springer, 578–591. 
5. Attneave, F. 1954. Some informational aspects of visual perception. Psychol. Rev. 61, 3, 183–193.
6. Averbuch, A., Schclar, A., and Donoho, D. 2005. Deblocking of block-transform compressed images using weighted sums of symmetrically aligned pixels. IEEE Trans. Image Process. 14, 2, 200–212. 
7. Barten, P. G. 1999. Contrast Sensitivity of the Human Eye and Its Effects on Image Quality. International Society for Optical Engineering.
8. Beaudot, W., and Mullen, K. 2003. How long range is contour integration in human color vision? In Visual Neuroscience, vol. 15, 51–64.
9. Bhat, P., Zitnick, C. L., Snavely, N., Agarwala, A., Agrawala, M., Curless, B., Cohen, M., and Kang, S. B. 2007. Using photographs to enhance videos of a static scene. In Proceedings of the Eurographics Symposium on Rendering Techniques. Eurographics, 327–338. 
10. Bhat, P., Curless, B., Cohen, M., and Zitnick, L. 2008. Fourier Analysis of the 2D screened poisson equation for gradient domain problems. In Proceedings of the European Conference on Computer Vision (ECCV’08). 
11. Bhat, P., Curless, B., Cohen, M., and Zitnick, L. 2008. Gradientshop: Gradient-domain image and video processing. http://www.GradientShop.com.
12. Black, M. J., Sapiro, G., Marimont, D. and Heeger, D., 1998. Robust anisotropic diffusion. IEEE Trans. Image Process. 
13. Castagno, R. and Ramponi, G. 1996. A rational filter for the removal of blocking artifacts in image sequences coded at low bitrate. In Proceedings of the European Signal Processing Conference (EUSIPC).
14. Drori, I., Leyvand, T., Fleishman, S., Cohen-Or, D., and Yeshurun., H. 2004. Video operations in the gradient domain. Tech. rep., Tel-Aviv University.
15. Elder, J. H., and Goldberg, R. M. 2001. Image editing in the contour domain. IEEE Trans. Pattern Anal. Mach. Intell. 23, 3, 291–296. 
16. Fattal, R., Lischinski, D., and Werman, M. 2002. Gradient domain high dynamic range compression. In Proceedings of the 29th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH’02). ACM Press, New York, 249–256. 
17. Freeman, W. T., and Adelson, E. H. 1991. The design and use of steerable filters. IEEE Trans. Pattern Anal. Mach. Intell. 13, 9, 891–906. 
18. Georgiev, T. 2006. Covariant derivatives and vision. In Proceedings of the 9th European Conference on Computer Vision (ECCV’06). 
19. Gooch, A. A., Olsen, S. C., Tumblin, J., and Gooch, B. 2005. Color2gray: Salience-preserving color removal. ACM Trans. Graph. 24, 3, 634–639. 
20. Hong, S., Chan, Y., and Siu, W. 1996. A practical real-time post-processing technique for block effect elimination. In Proceedings of the IEEE International Conference on Image Processing (ICIP’96). II: 21–24.
21. Itti, L., Koch, C., and Niebur, E. 1998. A model of saliency-based visual attention for rapid scene analysis. IEEE Trans. Pattern Anal. Mach. Intell. 
22. Kim, Tae-Hoon, Ahn, Jongwoo, Choi, and Gyu, M. 2007. Image dequantization: Restoration of quantized colors. Comput. Graph. Forum 26, 3, 619–626.
23. Levin, A., Zomet, A., Peleg, S., and Weiss, Y. 2004. Seamless image stitching in the gradient domain. In Hebrew University Tech. rep. 2003-82.
24. Levin, A., Lischinski, D., and Weiss, Y. 2004. Colorization using optimization. In ACM SIGGRAPH Papers (SIGGRAPH’04). ACM Press, New York, 689–694. 
25. Lischinski, D., Farbman, Z., Uyttendaele, M., and Szeliski, R. 2006. Interactive local adjustment of tonal values. In ACM SIGGRAPH Papers (SIGGRAPH’06). ACM Press, New York, 646–653. 
26. Marschner, S. R. and Greenberg, D. P. 1997. Inverse lighting for photography. In Proceedings of the 5th Color Imaging Conference. Society for Imaging Science and Technology.
27. McCann, J. and Pollard, N. S. 2008. Real-time gradient-domain painting. ACM Trans. Graph. 27, 3. 
28. Orzan, A., Bousseau, A., Barla, P., and Thollot, J. 2007. Structure-preserving manipulation of photographs. In Proceedings of the International Symposium on Non-Photorealistic Animation and Rendering (NPAR). 
29. Paris, S. 2008. Edge-preserving smoothing and mean-shift segmentation of video streams. In Proceedings of the 10th European Conference on Computer Vision (ECCV’08). Springer, 460–473. 
30. Pérez, P., Gangnet, M., and Blake, A. 2003. Poisson image editing. In ACM SIGGRAPH Papers (SIGGRAPH’03). ACM Press, New York, 313–318. 
31. Rempel, A. G., Trentacoste, M., Seetzen, H., Young, H. D., Heidrich, W., Whitehead, L., and Ward, G. 2007. Ldr2hdr: On-the-Fly reverse tone mapping of legacy video and photographs. ACM Trans. Graph. 26, 3, 39. 
32. Sand, P., and Teller, S. 2006. Particle video: Long-range motion estimation using point trajectories.In Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06). IEEE Computer Society, 2195–2202. 
33. Shewchuk, J. R. 1994. An introduction to the conjugate gradient method without the agonizing pain. http://www.cs.cmu.edu/~quake-papers/painless-conjugate-gradient.pdf.
34. Szeliski, R. 2006. Locally adapted hierarchical basis preconditioning. In ACM SIGGRAPH Papers (SIGGRAPH’06). ACM Press, New York, 1135–1143. 
35. Tomar, S. 2006. Converting video formats with ffmpeg. Linux J. 146, 10. 
36. Wang, H., Raskar, R., and Ahuja, N. 2004. Seamless video editing. In Proceedings of the 17th International Conference on Pattern Recognition, (ICPR’04) vol. 3, IEEE Computer Society, 858–861. 
37. Winnemöller, H., Olsen, S. C., and Gooch, B. 2006. Real-time video abstraction. ACM Trans. Graph. 25, 3, 1221–1226. 
38. Zeng, Y., Chen, W., and Peng, Q. 2006. A novel variational image model: Towards a unified approach to image editing. J. Comput. Sci. Technol.