“Video watercolorization using bidirectional texture advection” by Bousseau, Neyret, Thollot and Salesin

In this paper, we present a method for creating watercolor-like animation, starting from video as input. The method involves two main steps: applying textures that simulate a watercolor appearance; and creating a simplified, abstracted version of the video to which the texturing operations are applied. Both of these steps are subject to highly visible temporal artifacts, so the primary technical contributions of the paper are extensions of previous methods for texturing and abstraction to provide temporal coherence when applied to video sequences. To maintain coherence for textures, we employ texture advection along lines of optical flow. We furthermore extend previous approaches by incorporating advection in both forward and reverse directions through the video, which allows for minimal texture distortion, particularly in areas of disocclusion that are otherwise highly problematic. To maintain coherence for abstraction, we employ mathematical morphology extended to the temporal domain, using filters whose temporal extents are locally controlled by the degree of distortions in the optical flow. Together, these techniques provide the first practical and robust approach for producing watercolor animations from video, which we demonstrate with a number of examples.

1. Alp, B., Haavisto, P., Jarske, T., Oistamo, K., and Neuvo, Y. A. 1990. Median-based algorithms for image sequence processing. In SPIE Vol. 1360, Visual Communications and Image Processing, 122–134.Google Scholar
2. Bousseau, A., Kaplan, M., Thollot, J., and Sillion, F. X. 2006. Interactive watercolor rendering with temporal coherence and abstraction. In International Symposium on Non-Photorealistic Animation and Rendering (NPAR), 141 — 149. Google ScholarDigital Library
3. Chuang, Y.-Y., Agarwala, A., Curless, B., Salesin, D. H., and Szeliski, R. 2002. Video matting of complex scenes. ACM Transactions on Graphics (Proc. SIGGRAPH 2005) 21, 3 (July), 243–248. Google ScholarDigital Library
4. Collomosse, J. P., Rowntree, D., and Hall, P. M. 2005. Stroke surfaces: Temporally coherent artistic animations from video. IEEE Transactions on Visualization and Computer Graphics 11, 5 (Sept.), 540–549. Google ScholarDigital Library
5. Cunzi, M., Thollot, J., Paris, S., Debunne, G., Gascuel, J.-D., and Durand, F. 2003. Dynamic canvas for immersive non-photorealistic walkthroughs. In Graphics Interface, 121–130.Google Scholar
6. Curtis, C. J., Anderson, S. E., Seims, J. E., Fleischer, K. W., and Salesin, D. H. 1997. Computer-generated water-color. In SIGGRAPH 97, 421–430. Google ScholarDigital Library
7. Fang, H. 2006. Rototexture: Automated tools for texturing raw video. IEEE Transactions on Visualization and Computer Graphics 12, 6, 1580–1589. Google ScholarDigital Library
8. Haralick, R. M., Sternberg, S. R., and Zhuang, X. 1987. Image analysis using mathematical morphology. IEEE Trans. Pattern Anal. Mach. Intell. 9, 4, 532–550. Google ScholarDigital Library
9. Hays, J., and Essa, I. 2004. Image and video based painterly animation. In International Symposium on Non-Photorealistic Animation and Rendering (NPAR), 113–120. Google ScholarDigital Library
10. Hertzmann, A., and Perlin, K. 2000. Painterly rendering for video and interaction. In International Symposium on Non-Photorealistic Animation and Rendering (NPAR), 7–12. Google ScholarDigital Library
11. Horn, B. K. P. 1986. Robot Vision. MIT Press. ISBN 0-262-08159-8. Google ScholarDigital Library
12. Jobard, B., Erlebacher, G., and Hussaini, M. Y. 2001. Lagrangian-eulerian advection for unsteady flow visualization. In VIS ’01: Conference on Visualization ’01, 53–60. Google ScholarDigital Library
13. Johan, H., Hashimota, R., and Nishita, T. 2005. Creating watercolor style images taking into account painting techniques. Journal of the Society for Art and Science 3, 4, 207–215.Google ScholarCross Ref
14. Klein, A. W., Sloan, P.-P. J., Finkelstein, A., and Cohen, M. F. 2002. Stylized video cubes. In ACM-SIGGRAPH/EG Symposium on Computer Animation (SCA), 15–22. Google ScholarDigital Library
15. Kokaram, A. C. 1998. Motion Picture Restoration: Digital Algorithms for Artefact Suppression in Degraded Motion Picture Film and Video. Springer-Verlag. ISBN 3-540-76040-7. Google ScholarDigital Library
16. Laveau, N., and Bernard, C. 2005. Structuring elements following the optical flow. In Mathematical Morphology: 40 Years On, Proceedings of the 7th International Symposium on Mathematical Morphology, Springer-Verlag, Ed., 43–52.Google Scholar
17. Lei, E., and Chang, C.-F. 2004. Real-time rendering of water-color effects for virtual environments. In IEEE 2004 Pacific-Rim Conference on Multimedia, 474–481. Google ScholarDigital Library
18. Litwinowicz, P. C. 1997. Processing images and video for an impressionist effect. In SIGGRAPH 97, 407–414. Google ScholarDigital Library
19. Luft, T., and Deussen, O. 2006. Real-time watercolor illustrations of plants using a blurred depth test. In International Symposium on Non-Photorealistic Animation and Rendering (NPAR), 11–20. Google ScholarDigital Library
20. Lum, E. B., and Ma, K.-L. 2001. Non-photorealistic rendering using watercolor inspired textures and illumination. In Pacific Graphics, 322–331. Google ScholarDigital Library
21. Max, N., and Becker, B. 1995. Flow visualization using moving textures. In Proceedings of the ICASW/LaRC Symposium on Visualizing Time-Varying Data, 77–87.Google Scholar
22. Neyret, F. 2003. Advected textures. In ACM-SIGGRAPH/EG Symposium on Computer Animation (SCA), 147 — 153. Google ScholarDigital Library
23. Ozkan, M. K., Sezan, M. I., and Tekalp, A. M. 1993. Adaptive motion-compensated filtering of noisy image sequences. IEEE transactions on circuits and systems for video technology 3, 4, 277–290.Google Scholar
24. Sand, P., and Teller, S. 2006. Particle video: Long-range motion estimation using point trajectories. In CVPR, 2195 — 2202. Google ScholarDigital Library
25. Serra, J., and Vincent, L. 1992. An overview of morphological filtering. Circuits Syst. Signal Process. 11, 1, 47–108. Google ScholarDigital Library
26. Sims, K. 1992. Choreographed image flow. The Journal of Visualization and Computer Animation 3, 1, 31–43.Google ScholarCross Ref
27. Small, D. 1991. Modeling watercolor by simulating diffusion, pigment, and paper fibers. In SPIE, vol. 1460, 140–146.Google Scholar
28. Stam, J. 1999. Stable fluids. In SIGGRAPH 99, 121–128. Google ScholarDigital Library
29. Van Laerhoven, T., Liesenborgs, J., and Van Reeth, F. 2004. Real-time watercolor painting on a distributed paper model. In Computer Graphics International, 640–643. Google ScholarCross Ref
30. Wang, J. Y. A., and Adelson, E. H. 1994. Representing Moving Images with Layers. The IEEE Transaction on Image Processing Special Issue: Image Sequence Compression 3, 5, 625–638.Google ScholarDigital Library
31. Wang, J., Xu, Y., Shum, H.-Y., and Cohen, M. 2004. Video tooning. ACM Transactions on Graphics (proc. of SIGGRAPH 2004) 23, 3, 574 — 583. Google ScholarDigital Library
32. WEISS, B. 2006. Fast median and bilateral filtering. ACM Transactions on Graphics (proc. of SIGGRAPH 2006) 25, 3, 519–526. Google ScholarDigital Library
33. Winnemoller, H., Olsen, S. C., and Gooch, B. 2006. Realtime video abstraction. ACM Transactions on Graphics (proc. of SIGGRAPH 2006) 25, 3, 1221 — 1226. Google ScholarDigital Library