“Edge-preserving decompositions for multi-scale tone and detail manipulation” by Farbman, Fattal, Lischinski and Szeliski

Many recent computational photography techniques decompose an image into a piecewise smooth base layer, containing large scale variations in intensity, and a residual detail layer capturing the smaller scale details in the image. In many of these applications, it is important to control the spatial scale of the extracted details, and it is often desirable to manipulate details at multiple scales, while avoiding visual artifacts.In this paper we introduce a new way to construct edge-preserving multi-scale image decompositions. We show that current basedetail decomposition techniques, based on the bilateral filter, are limited in their ability to extract detail at arbitrary scales. Instead, we advocate the use of an alternative edge-preserving smoothing operator, based on the weighted least squares optimization framework, which is particularly well suited for progressive coarsening of images and for multi-scale detail extraction. After describing this operator, we show how to use it to construct edge-preserving multi-scale decompositions, and compare it to the bilateral filter, as well as to other schemes. Finally, we demonstrate the effectiveness of our edge-preserving decompositions in the context of LDR and HDR tone mapping, detail enhancement, and other applications.

1. Alvarez, L., Lions, P.-L., and Morel, J.-M. 1992. Image selective smoothing and edge detection by nonlinear diffusion. ii. SIAM Journal on Numerical Analysis 29, 3 (June), 845–866. Google ScholarDigital Library
2. Aubert, G., and Kornprobst, P. 2006. Mathematical Problems in Image Processing: Partial Differential Equations and the Calculus of Variations, vol. 147 of Applied Mathematical Sciences. Springer. Google ScholarDigital Library
3. Bae, S., Paris, S., and Durand, F. 2006. Two-scale tone management for photographic look. ACM Trans. Graph. 25, 3 (July), 654–662. Google ScholarDigital Library
4. Barash, D. 2002. A fundamental relationship between bilateral filtering, adaptive smoothing, and the nonlinear diffusion equation. IEEE Trans. Pattern Anal. Mach. Intell. 24, 6, 844–847. Google ScholarDigital Library
5. Black, M. J., Sapiro, G., Marimont, D. H., and Heeger, D. 1998. Robust anisotropic diffusion. IEEE Trans. Image Proc. 7, 3 (Mar.), 421–432. Google ScholarDigital Library
6. Buades, A., Coll, B., and Morel, J. M. 2006. The staircasing effect in neighborhood filters and its solution. IEEE Transactions on Image Processing 15, 6, 1499–1505. Google ScholarDigital Library
7. Buatois, L., Caumon, G., and Levy, B. 2007. Concurrent number cruncher: An efficient sparse linear solver on the GPU. In High Performance Computation Conference (HPCC), Springer. Google ScholarDigital Library
8. Burt, P., and Adelson, E. H. 1983. The Laplacian pyramid as a compact image code. IEEE Trans. Comm. 31, 532–540.Google ScholarCross Ref
9. Chen, J., Paris, S., and Durand, F. 2007. Real-time edge-aware image processing with the bilateral grid. ACM Trans. Graph. 26, 3 (July), Article 103. Google ScholarDigital Library
10. Choudhury, P., and Tumblin, J. 2003. The trilateral filter for high contrast images and meshes. In Proc. EGSR 2003, Eurographics, 186–196. Google ScholarDigital Library
11. Comaniciu, D., and Meer, P. 2002. Mean shift: A robust approach toward feature space analysis. IEEE Trans. Pattern Anal. Mach. Intell. 24, 5, 603–619. Google ScholarDigital Library
12. DeCarlo, D., and Santella, A. 2002. Stylization and abstraction of photographs. ACM Trans. Graph. 21, 3 (July), 769–776. Google ScholarDigital Library
13. Durand, F., and Dorsey, J. 2002. Fast bilateral filtering for the display of high-dynamic-range images. ACM Trans. Graph. 21, 3 (July), 257–266. Google ScholarDigital Library
14. Eisemann, E., and Durand, F. 2004. Flash photography enhancement via intrinsic relighting. ACM Trans. Graph. 23, 3 (August), 673–678. Google ScholarDigital Library
15. Elad, M. 2002. On the bilateral filter and ways to improve it. IEEE Trans. Image Proc. 11, 10, 1141–1151. Google ScholarDigital Library
16. Fattal, R., Lischinski, D., and Werman, M. 2002. Gradient domain high dynamic range compression. ACM Trans. Graph. 21, 3 (July), 249–256. Google ScholarDigital Library
17. Fattal, R., Agrawala, M., and Rusinkiewicz, S. 2007. Multiscale shape and detail enhancement from multi-light image collections. ACM Trans. Graph. 26, 3 (July), Article 51. Google ScholarDigital Library
18. Jobson, D. J., Rahman, Z., and Woodell, G. A. 1997. A multi-scale Retinex for bridging the gap between color images and the human observation of scenes. IEEE Trans. Image Proc. 6, 7 (July), 965–976. Google ScholarDigital Library
19. Khan, E. A., Reinhard, E., Fleming, R. W., and Bülthoff, H. H. 2006. Image-based material editing. ACM Trans. Graph. 25, 3 (July), 654–663. Google ScholarDigital Library
20. Lagendijk, R. L., Biemond, J., and Boekee, D. E. 1988. Regularized iterative image restoration with ringing reduction. IEEE Trans. Acoustics, Speech, and Signal Proc., Speech, Signal Proc. 36, 12 (December), 1874–1888.Google Scholar
21. Levin, A., Lischinski, D., and Weiss, Y. 2004. Colorization using optimization. ACM Trans. Graph. 23, 3 (August), 689–694. Google ScholarDigital Library
22. Levin, A., Fergus, R., Durand, F., and Freeman, W. T. 2007. Image and depth from a conventional camera with a coded aperture. ACM Trans. Graph. 26, 3 (July), Article 70. Google ScholarDigital Library
23. Li, Y., Sharan, L., and Adelson, E. H. 2005. Compressing and companding high dynamic range images with subband architectures. ACM Trans. Graph. 24, 3 (July), 836–844. Google ScholarDigital Library
24. Lischinski, D., Farbman, Z., Uyttendaele, M., and Szeliski, R. 2006. Interactive local adjustment of tonal values. ACM Trans. Graph. 25, 3 (July), 646–653. Google ScholarDigital Library
25. Mantiuk, R., Myszkowski, K., and Seidel, H.-P. 2006. A perceptual framework for contrast processing of high dynamic range images. ACM Trans. Appl. Percept. 3, 3, 286–308. Google ScholarDigital Library
26. Mrázek, P., Weickert, J., and Bruhn, A. 2006. On robust estimation and smoothing with spatial and tonal kernels. In Geometric Properties from Incomplete Data, R. Klette, R. Kozera, L. Noakes, and J. Weickert, Eds. Springer, Dordrecht, 335–352.Google Scholar
27. Nordstrom, K. N. 1989. Biased anisotropic diffusion — a unified regularization and diffusion approach to edge detection. Tech. Rep. UCB/CSD-89-514, EECS Department, University of California, Berkeley, May. Google ScholarDigital Library
28. Oh, B. M., Chen, M., Dorsey, J., and Durand, F. 2001. Image-based modeling and photo editing. In Proc. ACM SIGGRAPH 2001, ACM, E. Fiume, Ed., 433–442. Google ScholarDigital Library
29. Oppenheim, A. V., and Schafer, R. W. 1989. Discrete-Time Signal Processing. Prentice Hall. Google ScholarDigital Library
30. Paris, S., and Durand, F. 2006. A fast approximation of the bilateral filter using a signal processing approach. In Proc. ECCV ’06, IV: 568–580. Google ScholarDigital Library
31. Paris, S. 2007. A gentle introduction to bilateral filtering and its applications. In ACM SIGGRAPH 2007 courses, Course 13. Google ScholarDigital Library
32. Pattanaik, S. N., Ferwerda, J. A., Fairchild, M. D., and Greenberg, D. P. 1998. A multiscale model of adaptation and spatial vision for realistic image display. In Proc. ACM SIGGRAPH 98, M. Cohen, Ed., 287–298. Google ScholarDigital Library
33. Perona, P., and Malik, J. 1990. Scale-space and edge detection using anisotropic diffusion. IEEE Trans. Pattern Anal. Machine Intell. 12, 7 (July), 629–639. Google ScholarDigital Library
34. Petschnigg, G., Szeliski, R., Agrawala, M., Cohen, M., Hoppe, H., and Toyama, K. 2004. Digital photography with flash and no-flash image pairs. ACM Trans. Graph. 23, 3 (August), 664–672. Google ScholarDigital Library
35. Reinhard, E., Stark, M., Shirley, P., and Ferwerda, J. 2002. Photographic tone reproduction for digital images. ACM Trans. Graph. 21, 3 (July), 267–276. Google ScholarDigital Library
36. Saad, Y. 2003. Iterative Methods for Sparse Linear Systems, second ed. SIAM. Google ScholarDigital Library
37. Scales, J. A., and Gersztenkorn, A. 1988. Robust methods in inverse theory. Inverse Problems 4, 1071–1091.Google ScholarCross Ref
38. Scherzer, O., and Weickert, J. 2000. Relations between regularization and diffusion filtering. Journal of Mathematical Imaging and Vision 12, 1 (February), 43–63. Google ScholarDigital Library
39. Schlick, C. 1994. Quantization techniques for visualization of high dynamic range pictures. In Photorealistic Rendering Techniques, Springer-Verlag, P. Shirley, G. Sakas, and S. Müller, Eds., 7–20.Google Scholar
40. Tomasi, C., and Manduchi, R. 1998. Bilateral filtering for gray and color images. In Proc. ICCV ’98, IEEE Computer Society, 839–846. Google ScholarDigital Library
41. Tumblin, J., and Turk, G. 1999. LCIS: A boundary hierarchy for detail-preserving contrast reduction. In Proc. ACM SIGGRAPH 99, A. Rockwood, Ed., ACM, 83–90. Google ScholarDigital Library
42. Weiss, B. 2006. Fast median and bilateral filtering. ACM Trans. Graph. 25, 3 (July), 519–526. Google ScholarDigital Library
43. Winnemöller, H., Olsen, S. C., and Gooch, B. 2006. Realtime video abstraction. ACM Trans. Graph. 25, 3 (July), 1221–1226. Google ScholarDigital Library
44. Zervakis, M. E. 1990. Nonlinear image restoration techniques. PhD thesis, Univ. Toronto, Toronto, ON, Canada.Google Scholar