“Multi-scale image harmonization” by Sunkavalli, Johnson, Matusik and Pfister

Traditional image compositing techniques, such as alpha matting and gradient domain compositing, are used to create composites that have plausible boundaries. But when applied to images taken from different sources or shot under different conditions, these techniques can produce unrealistic results. In this work, we present a framework that explicitly matches the visual appearance of images through a process we call image harmonization, before blending them. At the heart of this framework is a multi-scale technique that allows us to transfer the appearance of one image to another. We show that by carefully manipulating the scales of a pyramid decomposition of an image, we can match contrast, texture, noise, and blur, while avoiding image artifacts. The output composite can then be reconstructed from the modified pyramid coefficients while enforcing both alpha-based and seamless boundary constraints. We show how the proposed framework can be used to produce realistic composites with minimal user interaction in a number of different scenarios.

1. Agarwala, A., Dontcheva, M., Agrawala, M., Drucker, S., Colburn, A., Curless, B., Salesin, D., and Cohen, M. 2004. Interactive digital photomontage. ACM Transactions on Graphics 23, 3, 294–302. Google ScholarDigital Library
2. Agarwala, A. 2007. Efficient gradient-domain compositing using quadtrees. ACM Transactions on Graphics 26, 3, 94. Google ScholarDigital Library
3. Bae, S., Paris, S., and Durand, F. 2006. Two-scale tone management for photographic look. ACM Transactions on Graphics 25, 3, 637–645. Google ScholarDigital Library
4. Burt, P. J., and Adelson, E. H. 1983. The Laplacian pyramid as a compact image code. IEEE Transactions on Communications COM-31, 4, 532–540.Google ScholarCross Ref
5. Burt, P. J., and Adelson, E. H. 1983. A multiresolution spline with application to image mosaics. ACM Transactions on Graphics 2, 4, 217–236. Google ScholarDigital Library
6. Chen, J., Tang, C.-K., and Wang, J. 2009. Noise brush: interactive high quality image-noise separation. ACM Transactions on Graphics 28, 5, 1–10. Google ScholarDigital Library
7. Cohen-Or, D., Sorkine, O., Gal, R., Leyvand, T., and Xu, Y.-Q. 2006. Color harmonization. ACM Transactions on Graphics 25, 3, 624–630. Google ScholarDigital Library
8. Davis, T. A. 2006. Direct Methods for Sparse Linear Systems (Fundamentals of Algorithms 2). Society for Industrial and Applied Mathematics, Philadelphia, PA, USA. Google ScholarDigital Library
9. Farbman, Z., Fattal, R., Lischinski, D., and Szeliski, R. 2008. Edge-preserving decompositions for multi-scale tone and detail manipulation. ACM Transactions on Graphics 27, 3, 1–10. Google ScholarDigital Library
10. Farbman, Z., Hoffer, G., Lipman, Y., Cohen-Or, D., and Lischinski, D. 2009. Coordinates for instant image cloning. ACM Transactions on Graphics 28, 3, 1–9. Google ScholarDigital Library
11. Fattal, R., Agrawala, M., and Rusinkiewicz, S. 2007. Multiscale shape and detail enhancement from multi-light image collections. ACM Transactions on Graphics 26, 3, 51. Google ScholarDigital Library
12. Georgiev, T. 2004. Photoshop healing brush: a tool for seamless cloning. In Workshop on Applications of Computer Vision (ECCV 2004), 1–8.Google Scholar
13. Gonzalez, R. C., and Woods, R. E. 2001. Digital Image Processing. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA. Google ScholarDigital Library
14. Heeger, D. J., and Bergen, J. R. 1995. Pyramid-based texture analysis/synthesis. SIGGRAPH ’95: Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, 229–238. Google ScholarDigital Library
15. Jia, J., Sun, J., Tang, C.-K., and Shum, H.-Y. 2006. Drag-and-drop pasting. ACM Transactions on Graphics 25, 3, 631–637. Google ScholarDigital Library
16. Lalonde, J.-F., and Efros, A. A. 2007. Using color compatibility for assessing image realism. In IEEE International Conference on Computer Vision.Google Scholar
17. Levin, A., Zomet, A., Peleg, S., and Weiss, Y. 2004. Seamless image stitching in the gradient domain. In European Conference on Computer Vision.Google Scholar
18. Li, Y., Sharan, L., and Adelson, E. H. 2005. Compressing and companding high dynamic range images with subband architectures. ACM Transactions on Graphics 24, 3, 836–844. Google ScholarDigital Library
19. McCann, J., and Pollard, N. S. 2008. Real-time gradient-domain painting. ACM Transactions on Graphics 27, 3, 1–7. Google ScholarDigital Library
20. Pérez, P., Gangnet, M., and Blake, A. 2003. Poisson image editing. ACM Transactions on Graphics 22, 3, 313–318. Google ScholarDigital Library
21. Pitie, F., Kokaram, A. C., and Dahyot, R. 2005. N-dimensional probablility density function transfer and its application to colour transfer. In IEEE International Conference on Computer Vision. Google ScholarDigital Library
22. Porter, T., and Duff, T. 1984. Compositing digital images. In SIGGRAPH ’84: Proceedings of the 11th annual conference on Computer graphics and interactive techniques, 253–259. Google ScholarDigital Library
23. Portilla, J., and Simoncelli, E. P. 2000. A parametric texture model based on joint statistics of complex wavelet coefficients. International Journal of Computer Vision 40, 1, 49–70. Google ScholarDigital Library
24. Reinhard, E., Ashikhmin, M., Gooch, B., and Shirley, P. 2001. Color transfer between images. IEEE Computer Graphics and Applications 21, 5, 34–41. Google ScholarDigital Library
25. Sun, J., Jia, J., Tang, C.-K., and Shum, H.-Y. 2004. Poisson matting. ACM Transactions on Graphics 23, 3, 315–321. Google ScholarDigital Library
26. Szeliski, R. 2006. Locally adapted hierarchical basis preconditioning. ACM Transactions on Graphics 25, 3, 1135–1143. Google ScholarDigital Library
27. Tomasi, C., and Manduchi, R. 1998. Bilateral filtering for gray and color images. In IEEE International Conference on Computer Vision. Google ScholarDigital Library
28. Wang, J., Agrawala, M., and Cohen, M. F. 2007. Soft scissors: an interactive tool for realtime high quality matting. ACM Transactions on Graphics 26, 3, 9. Google ScholarDigital Library