“Example-based video color grading” by Bonneel, Sunkavalli, Paris and Pfister
Conference:
Type(s):
Title:
- Example-based video color grading
Session/Category Title: Color & Compositing
Presenter(s)/Author(s):
Moderator(s):
Abstract:
In most professional cinema productions, the color palette of the movie is painstakingly adjusted by a team of skilled colorists — through a process referred to as color grading — to achieve a certain visual look. The time and expertise required to grade a video makes it difficult for amateurs to manipulate the colors of their own video clips. In this work, we present a method that allows a user to transfer the color palette of a model video clip to their own video sequence. We estimate a per-frame color transform that maps the color distributions in the input video sequence to that of the model video clip. Applying this transformation naively leads to artifacts such as bleeding and flickering. Instead, we propose a novel differential-geometry-based scheme that interpolates these transformations in a manner that minimizes their curvature, similarly to curvature flows. In addition, we automatically determine a set of keyframes that best represent this interpolated transformation curve, and can be used subsequently, to manually refine the color grade. We show how our method can successfully transfer color palettes between videos for a range of visual styles and a number of input video clips.
References:
1. Amari, S., and Nagaoka, H. 2000. Methods of Information Geometry, vol. 191 of Translations of Mathematical monographs. Oxford University Press.Google Scholar
2. An, X., and Pellacini, F. 2010. User-controllable color transfer. Computer Graphics Forum 29, 2, 263–271.Google ScholarCross Ref
3. Bae, S., Paris, S., and Durand, F. 2006. Two-scale tone management for photographic look. ACM Trans. on Graphics (Proc. of ACM SIGGRAPH 2006) 25, 3, 637–645. Google ScholarDigital Library
4. Bai, X., Wang, J., Simons, D., and Sapiro, G. 2009. Video SnapCut: robust video object cutout using localized classifiers. In ACM Trans. on Graphics (Proc. of ACM SIGGRAPH 2009), 70:1–70:11. Google ScholarDigital Library
5. Bonneel, N., van de Panne, M., Paris, S., and Heidrich, W. 2011. Displacement interpolation using lagrangian mass transport. In ACM Trans. on Graphics (Proc. of ACM SIGGRAPH Asia 2011), 158:1–158:12. Google ScholarDigital Library
6. Chang, Y., Saito, S., and Nakajima, M. 2007. Example-based color transformation of image and video using basic color categories. Image Processing, IEEE Trans. on 16, 2, 329–336. Google ScholarDigital Library
7. Devroye, L. 1986. Non-Uniform Random Variate Generation. Springer-Verlag. Section 2.2. Inversion by numerical solution of F(X) = U.Google Scholar
8. do Carmo, M. P. 1992. Riemannian Geometry. Springer, Jan.Google Scholar
9. Farbman, Z., and Lischinski, D. 2011. Tonal stabilization of video. ACM Trans. on Graphics (Proc. of ACM SIGGRAPH 2011) 30, 4, 89:1–89:9. Google ScholarDigital Library
10. Ferradans, S., Xia, G.-S., Peyré, G., and Aujol, J.-F. 2012. Optimal transport mixing of gaussian texture models. Tech. rep., Preprint Hal-00662720.Google Scholar
11. Galassi, M., Davis, J., Gough, B., Jungman, G., Booth, M., and Rossi, F., 2011. GNU scientific library – reference manual, version 1.15, sec. 21.7 weighted samples.Google Scholar
12. Gastal, E. S. L., and Oliveira, M. M. 2011. Domain transform for edge-aware image and video processing. In ACM Trans. on Graphics (Proc. of ACM SIGGRAPH 2011), 69:1–69:12. Google ScholarDigital Library
13. HaCohen, Y., Shechtman, E., Goldman, D. B., and Lischinski, D. 2011. Non-rigid dense correspondence with applications for image enhancement. ACM Transactions on Graphics (Proceedings of ACM SIGGRAPH 2011) 30, 4, 70:1–70:9. Google ScholarDigital Library
14. Ilmanen, T. 1995. Lectures on mean curvature flow and related equations. In Lecture Notes, ICTP, Trieste.Google Scholar
15. Johnson, M. K., Dlale, K., Avidan, S., Pfister, H., Freeman, W. T., and Matusik, W. 2011. CG2Real: improving the realism of computer generated images using a large collection of photographs. IEEE Trans. on Visualization and Computer Graphics 17, 9 (Sept.), 1273–1285. Google ScholarDigital Library
16. Kagarlitsky, S., Moses, Y., and Hel-Or, Y. 2009. Piecewise-consistent color mappings of images acquired under various conditions. In Computer Vision, 2009 IEEE 12th International Conference on, 2311–2318.Google Scholar
17. Kiser, C., Reinhard, E., Tocci, M., and Tocci, N. 2012. Real time automated tone mapping system for HDR video. In Proc. of the IEEE Int. Conference on Image Processing, IEEE.Google Scholar
18. Lang, M., Wang, O., Aydin, T., Smolic, A., and Gross, M. 2012. Practical temporal consistency for image-based graphics applications. ACM Trans. on Graphics (Proc. of ACM SIGGRAPH 2012) 31, 4 (July), 34:1–34:8. Google ScholarDigital Library
19. Levin, A., Lischinski, D., and Weiss, Y. 2008. A closed-form solution to natural image matting. IEEE Trans. on Pattern Analysis and Machine Intelligence 30, 2 (Feb.), 228–242. Google ScholarDigital Library
20. Murray, N., Skaff, S., Marchesotti, L., and Perronnin, F. 2011. Towards automatic concept transfer. In Proc. of the Eurographics Symposium on Non-Photorealistic Animation and Rendering, ACM, NPAR ’11, 167–176. Google ScholarDigital Library
21. Oldenborg, M. 2006. A comparison between techniques for color grading in games. PhD thesis, University of Skövde, School of Humanities and Informatics.Google Scholar
22. Oskam, T., Hornung, A., Sumner, R. W., and Gross, M. 2012. Fast and stable color balancing for images and augmented reality. In 2nd Int. Conf. on 3D Imaging, Modeling, Processing, Visualization and Transmission (3DIMPVT), 49–56. Google ScholarDigital Library
23. Paris, S. 2008. Edge-preserving smoothing and mean-shift segmentation of video streams. In Proc. of the 10th European Conference on Computer Vision, ECCV ’08, 460–473. Google ScholarDigital Library
24. Park, H.-S., and Jun, C.-H. 2009. A simple and fast algorithm for K-medoids clustering. Expert Systems with Applications 36, 2, Part 2 (Mar.), 3336–3341. Google ScholarDigital Library
25. Pitié, F., and Kokaram, A. C. 2007. The linear mongekantorovitch linear colour mapping for example-based colour transfer. In 4th European Conference on Visual Media Production, 2007. IETCVMP, 1–9.Google ScholarCross Ref
26. Pitié, F., Kokaram, A. C., and Dahyot, R. 2005. N-dimensional probablility density function transfer and its application to colour transfer. In Proceedings of the Tenth IEEE Int. Conf. on Computer Vision – Vol. 2, ICCV ’05, 1434–1439. Google ScholarDigital Library
27. Pouli, T., and Reinhard, E. 2011. Progressive color transfer for images of arbitrary dynamic range. Computers & Graphics 35, 1 (Feb.), 67–80. Google ScholarDigital Library
28. Rabin, J., Delon, J., and Gousseau, Y. 2010. Regularization of transportation maps for color and contrast transfer. In 2010 17th IEEE Int. Conf. on Image Processing (ICIP), 1933–1936.Google Scholar
29. Reinhard, E., and Pouli, T. 2011. Colour spaces for colour transfer. In Proceedings of the Third international conference on Computational color imaging, Springer-Verlag, Berlin, Heidelberg, CCIW’11, 1–15. Google ScholarDigital Library
30. Reinhard, E., Ashikhmin, M., Gooch, B., and Shirley, P. 2001. Color transfer between images. IEEE Comput. Graph. Appl. 21, 5 (Sept.), 34–41. Google ScholarDigital Library
31. Selan, J. 2012. Cinematic color: From your monitor to the big screen. Visual Effects Soc. Tech. Committee White Paper (Oct.).Google Scholar
32. Tai, Y.-W., Jia, J., and Tang, C.-K. 2005. Local color transfer via probabilistic segmentation by expectation-maximization. In Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on, vol. 1, 747–754 vol. 1. Google ScholarDigital Library
33. Takatsu, A. 2011. Wasserstein geometry of gaussian measures. Osaka Journal of Mathematics 48, 4 (Dec.), 1005–1026. Mathematical Reviews number (MathSciNet): MR2648273.Google Scholar
34. Villani, C. 2003. Topics in Optimal Transportation. Graduate Studies in Mathematics Series. Amer Mathematical Society.Google Scholar
35. Wang, B., Yu, Y., and Xu, Y.-Q. 2011. Example-based image color and tone style enhancement. In ACM Trans. on Graphics (Proc. of ACM SIGGRAPH 2011), 64:1–64:12. Google ScholarDigital Library
36. Xue, S., Agarwala, A., Dorsey, J., and Rushmeier, H. E. 2012. Understanding and improving the realism of image composites. ACM Trans. on Graphics (Proc. of ACM SIGGRAPH 2012) 31, 4, 84:1–84:10. Google ScholarDigital Library
37. Yatziv, L., and Sapiro, G. 2006. Fast image and video colorization using chrominance blending. IEEE Trans. on Image Processing 15, 5 (May), 1120–1129. Google ScholarDigital Library
