“Decomposing time-lapse paintings into layers” by Tan, Dvorožňák, Sýkora and Gingold

The creation of a painting, in the physical world or digitally, is a process that occurs over time. Later strokes cover earlier strokes, and strokes painted at a similar time are likely to be part of the same object. In the final painting, this temporal history is lost, and a static arrangement of color is all that remains. The rich literature for interacting with image editing history cannot be used. To enable these interactions, we present a set of techniques to decompose a time lapse video of a painting (defined generally to include pencils, markers, etc.) into a sequence of translucent “stroke” images. We present translucency-maximizing solutions for recovering physical (Kubelka and Munk layering) or digital (Porter and Duff “over” blending operation) paint parameters from before/after image pairs. We also present a pipeline for processing real-world videos of paintings capable of handling long-term occlusions, such as the painter’s hand and its shadow, color shifts, and noise.

1. Amati, C., and Brostow, G. J. 2010. Modeling 2.5D plants from ink paintings. In Proceedings of Eurographics Symposium on Sketch-Based Interfaces and Modeling Symposium, 41–48. Google ScholarDigital Library
2. Barbarić -Mikočević, V. D.-M. Ž., and Itrić, K. 2011. Kubelka-Munk theory in describing optical properties of paper (i). Technical Gazette 18, 1, 117–124.Google Scholar
3. Baxter, W. V., Wendt, J., and Lin, M. C. 2004. IMPaSTo: A realistic, interactive model for paint. In Proceedings of the International Symposium on Non-Photorealistic Animation and Rendering, 45–56. Google ScholarDigital Library
4. Berthouzoz, F., Li, W., Dontcheva, M., and Agrawala, M. 2011. A framework for content-adaptive photo manipulation macros: Application to face, landscape, and global manipulations. ACM Transactions on Graphics 30, 5, 120. Google ScholarDigital Library
5. Bonanni, L., Xiao, X., Hockenberry, M., Subramani, P., Ishii, H., Seracini, M., and Schulze, J. 2009. Wetpaint: Scraping through multi-layered images. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 571–574. Google ScholarDigital Library
6. Budsberg, J. B. 2007. Pigmented Colorants: Dependency on Media and Time. Master’s thesis, Cornell Univrsity, Ithaca, New York, USA.Google Scholar
7. Chen, H.-T., Wei, L.-Y., and Chang, C.-F. 2011. Nonlinear revision control for images. ACM Transactions on Graphics 30, 4, 105. Google ScholarDigital Library
8. Chen, T., Wei, L.-Y., Hartmann, B., and Agrawala, M. 2012. Data-driven history list for image editing. Tech. Rep. TR-2012-07, The University of Hong Kong.Google Scholar
9. Chen, H.-T., Grossman, T., Wei, L.-Y., Schmidt, R. M., Hartmann, B., Fitzmaurice, G., and Agrawala, M. 2014. History assisted view authoring for 3D models. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2027–2036. Google ScholarDigital Library
10. Cheung, S.-c. S., and Kamath, C. 2004. Robust techniques for background subtraction in urban traffic video. In Proceedings of SPIE, vol. 5308, 881–892.Google Scholar
11. Curtis, C. J., Anderson, S. E., Seims, J. E., Fleischer, K. W., and Salesin, D. H. 1997. Computer-generated watercolor. In ACM SIGGRAPH Conference Proceedings, 421–430. Google ScholarDigital Library
12. Dannen, C., 2012. The magical tech behind Paper for iPad’s color-mixing perfection, Nov. Accessed: January 10th, 2015.Google Scholar
13. Denning, J. D., and Pellacini, F. 2013. MeshGit: Diffing and merging meshes for polygonal modeling. ACM Transactions on Graphics 32, 4, 35. Google ScholarDigital Library
14. Farid, H., and Adelson, E. H. 1999. Separating reflections from images by use of independent component analysis. Journal of the Optical Society of America A 16, 9, 2136–2145.Google ScholarCross Ref
15. Fu, H., Zhou, S., Liu, L., and Mitra, N. J. 2011. Animated construction of line drawings. ACM Transactions on Graphics 30, 6, 133. Google ScholarDigital Library
16. Godbehere, A., Matsukawa, A., and Goldberg, K. 2012. Visual tracking of human visitors under variable-lighting conditions for a responsive audio art installation. In Proceedings of American Control Conference, 4305–4312.Google Scholar
17. Grabler, F., Agrawala, M., Li, W., Dontcheva, M., and Igarashi, T. 2009. Generating photo manipulation tutorials by demonstration. ACM Transactions on Graphics 28, 3, 66. Google ScholarDigital Library
18. Grossman, T., Matejka, J., and Fitzmaurice, G. 2010. Chronicle: Capture, exploration, and playback of document workflow histories. In Proceedings of ACM Symposium on User Interface Software and Technology, 143–152. Google ScholarDigital Library
19. Haase, C. S., and Meyer, G. W. 1992. Modeling pigmented materials for realistic image synthesis. ACM Transactions on Graphics 11, 4, 305–335. Google ScholarDigital Library
20. Hu, S.-M., Xu, K., Ma, L.-Q., Liu, B., Jiang, B.-Y., and Wang, J. 2013. Inverse image editing: Recovering a semantic editing history from a before-and-after image pair. ACM Transactions on Graphics 32, 6, 194. Google ScholarDigital Library
21. Hubbe, M. A., Pawlak, J. J., and Koukoulas, A. A. 2008. Paper’s appearance: A review. BioResources 3, 2, 627–665.Google ScholarCross Ref
22. Karsch, K., Golparvar-Fard, M., and Forsyth, D. 2014. Constructaide: Analyzing and visualizing construction sites through photographs and building models. ACM Transactions on Graphics 33, 6, 176. Google ScholarDigital Library
23. Konieczny, J., and Meyer, G. 2009. Airbrush simulation for artwork and computer modeling. In Proceedings of International Symposium on Non-Photorealistic Animation and Rendering, 61–69. Google ScholarDigital Library
24. Kubelka, P., and Munk, F. 1931. An article on optics of paint layers. Zeitschrift für Technische Physik 12, 593–601.Google Scholar
25. Kubelka, P. 1948. New contributions to the optics of intensely light-scattering materials. Part I. Journal of the Optical Society of America 38, 5, 448–448.Google ScholarCross Ref
26. Kubelka, P. 1954. New contributions to the optics of intensely light-scattering materials. Part II: Nonhomogeneous layers. Journal of the Optical Society of America 44, 4, 330–334.Google ScholarCross Ref
27. Lu, J., DiVerdi, S., Chen, W. A., Barnes, C., and Finkelstein, A. 2014. RealPigment: Paint compositing by example. In Proceedings of International Symposium on Non-Photorealistic Animation and Rendering, 21–30. Google ScholarDigital Library
28. Matzen, K., and Snavely, N. 2014. Scene chronology. In Proceedings of European Conference on Computer Vision. 615–630.Google Scholar
29. McCann, J., and Pollard, N. 2009. Local layering. ACM Transactions on Graphics 28, 3, 84. Google ScholarDigital Library
30. McCann, J., and Pollard, N. 2012. Soft stacking. Computer Graphics Forum 31, 2, 469–478. Google ScholarDigital Library
31. Nancel, M., and Cockburn, A. 2014. Causality: A conceptual model of interaction history. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 1777–1786. Google ScholarDigital Library
32. Noris, G., Sýkora, D., Shamir, A., Coros, S., Whited, B., Simmons, M., Hornung, A., Gross, M., and Sumner, R. 2012. Smart scribbles for sketch segmentation. Computer Graphics Forum 31, 8, 2516–2527. Google ScholarDigital Library
33. Novick, L. R., and Tversky, B. 1987. Cognitive constraints on ordering operations: The case of geometric analogies. Journal of Experimental Psychology: General 116, 1, 50.Google ScholarCross Ref
34. Porter, T., and Duff, T. 1984. Compositing digital images. ACM SIGGRAPH Computer Graphics 18, 3, 253–259. Google ScholarDigital Library
35. Richardt, C., Lopez-Moreno, J., Bousseau, A., Agrawala, M., and Drettakis, G. 2014. Vectorising bitmaps into semi-transparent gradient layers. Computer Graphics Forum (Proceedings of EGSR) 33, 4 (July), 11–19. Google ScholarDigital Library
36. Rubinstein, M., Liu, C., Sand, P., Durand, F., and Freeman, W. T. 2011. Motion denoising with application to time-lapse photography. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 313–320. Google ScholarDigital Library
37. Smith, A. R., and Blinn, J. F. 1996. Blue screen matting. In ACM SIGGRAPH Conference Proceedings, 259–268. Google ScholarDigital Library
38. Su, S. L., Paris, S., and Durand, F. 2009. QuickSelect: History-based selection expansion. In Proceedings of Graphics Interface, 215–221. Google ScholarDigital Library
39. Szeliski, R., Avidan, S., and Anandan, P. 2000. Layer extraction from multiple images containing reflections and transparency. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, vol. 1, 246–253.Google Scholar
40. Tange, O. 2011. GNU Parallel: The command-line power tool. ;login: The USENIX Magazine 36, 1 (Feb), 42–47.Google Scholar
41. Taylor, H. A., and Tversky, B. 1992. Descriptions and depictions of environments. Memory & Cognition 20, 5, 483–496.Google ScholarCross Ref
42. Tomasi, C., and Manduchi, R. 1998. Bilateral filtering for gray and color images. In Proceedings of IEEE International Conference on Computer Vision, 839–846. Google ScholarDigital Library
43. Tversky, B. 1999. What does drawing reveal about thinking? Visual and Spatial Reasoning in Design, 93–101.Google Scholar
44. van Sommers, P. 1984. Drawing and cognition: Descriptive and experimental studies of graphic production processes. Cambridge University Press.Google Scholar
45. VisTrails, I., 2009. Vistrails provenance explorer for maya. http://www.vistrails.com/maya.html.Google Scholar
46. Xing, J., Chen, H.-T., and Wei, L.-Y. 2014. Autocomplete painting repetitions. ACM Transactions on Graphics 33, 6, 172. Google ScholarDigital Library
47. Xu, S., Xu, Y., Kang, S. B., Salesin, D. H., Pan, Y., and Shum, H.-Y. 2006. Animating chinese paintings through stroke-based decomposition. ACM Transactions on Graphics 25, 2, 239–267. Google ScholarDigital Library
48. Xu, L., Lu, C., Xu, Y., and Jia, J. 2011. Image smoothing via L0 gradient minimization. ACM Transactions on Graphics 30, 6, 174. Google ScholarDigital Library
49. Zivkovic, Z., and van der Heijden, F. 2006. Efficient adaptive density estimation per image pixel for the task of background subtraction. Pattern Recognition Letters 27, 7, 773–780. Google ScholarDigital Library
50. Zongker, D. E., Werner, D. M., Curless, B., and Salesin, D. H. 1999. Environment matting and compositing. In ACM SIGGRAPH Conference Proceedings, 205–214. Google ScholarDigital Library