“Decomposing time-lapse paintings into layers” by Tan, Dvorožňák, Sýkora and Gingold

  • ©Jianchao Tan, Marek Dvorožňák, Daniel Sýkora, and Yotam Gingold




    Decomposing time-lapse paintings into layers



    The creation of a painting, in the physical world or digitally, is a process that occurs over time. Later strokes cover earlier strokes, and strokes painted at a similar time are likely to be part of the same object. In the final painting, this temporal history is lost, and a static arrangement of color is all that remains. The rich literature for interacting with image editing history cannot be used. To enable these interactions, we present a set of techniques to decompose a time lapse video of a painting (defined generally to include pencils, markers, etc.) into a sequence of translucent “stroke” images. We present translucency-maximizing solutions for recovering physical (Kubelka and Munk layering) or digital (Porter and Duff “over” blending operation) paint parameters from before/after image pairs. We also present a pipeline for processing real-world videos of paintings capable of handling long-term occlusions, such as the painter’s hand and its shadow, color shifts, and noise.


    1. Amati, C., and Brostow, G. J. 2010. Modeling 2.5D plants from ink paintings. In Proceedings of Eurographics Symposium on Sketch-Based Interfaces and Modeling Symposium, 41–48. Google ScholarDigital Library
    2. Barbarić -Mikočević, V. D.-M. Ž., and Itrić, K. 2011. Kubelka-Munk theory in describing optical properties of paper (i). Technical Gazette 18, 1, 117–124.Google Scholar
    3. Baxter, W. V., Wendt, J., and Lin, M. C. 2004. IMPaSTo: A realistic, interactive model for paint. In Proceedings of the International Symposium on Non-Photorealistic Animation and Rendering, 45–56. Google ScholarDigital Library
    4. Berthouzoz, F., Li, W., Dontcheva, M., and Agrawala, M. 2011. A framework for content-adaptive photo manipulation macros: Application to face, landscape, and global manipulations. ACM Transactions on Graphics 30, 5, 120. Google ScholarDigital Library
    5. Bonanni, L., Xiao, X., Hockenberry, M., Subramani, P., Ishii, H., Seracini, M., and Schulze, J. 2009. Wetpaint: Scraping through multi-layered images. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 571–574. Google ScholarDigital Library
    6. Budsberg, J. B. 2007. Pigmented Colorants: Dependency on Media and Time. Master’s thesis, Cornell Univrsity, Ithaca, New York, USA.Google Scholar
    7. Chen, H.-T., Wei, L.-Y., and Chang, C.-F. 2011. Nonlinear revision control for images. ACM Transactions on Graphics 30, 4, 105. Google ScholarDigital Library
    8. Chen, T., Wei, L.-Y., Hartmann, B., and Agrawala, M. 2012. Data-driven history list for image editing. Tech. Rep. TR-2012-07, The University of Hong Kong.Google Scholar
    9. Chen, H.-T., Grossman, T., Wei, L.-Y., Schmidt, R. M., Hartmann, B., Fitzmaurice, G., and Agrawala, M. 2014. History assisted view authoring for 3D models. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2027–2036. Google ScholarDigital Library
    10. Cheung, S.-c. S., and Kamath, C. 2004. Robust techniques for background subtraction in urban traffic video. In Proceedings of SPIE, vol. 5308, 881–892.Google Scholar
    11. Curtis, C. J., Anderson, S. E., Seims, J. E., Fleischer, K. W., and Salesin, D. H. 1997. Computer-generated watercolor. In ACM SIGGRAPH Conference Proceedings, 421–430. Google ScholarDigital Library
    12. Dannen, C., 2012. The magical tech behind Paper for iPad’s color-mixing perfection, Nov. Accessed: January 10th, 2015.Google Scholar
    13. Denning, J. D., and Pellacini, F. 2013. MeshGit: Diffing and merging meshes for polygonal modeling. ACM Transactions on Graphics 32, 4, 35. Google ScholarDigital Library
    14. Farid, H., and Adelson, E. H. 1999. Separating reflections from images by use of independent component analysis. Journal of the Optical Society of America A 16, 9, 2136–2145.Google ScholarCross Ref
    15. Fu, H., Zhou, S., Liu, L., and Mitra, N. J. 2011. Animated construction of line drawings. ACM Transactions on Graphics 30, 6, 133. Google ScholarDigital Library
    16. Godbehere, A., Matsukawa, A., and Goldberg, K. 2012. Visual tracking of human visitors under variable-lighting conditions for a responsive audio art installation. In Proceedings of American Control Conference, 4305–4312.Google Scholar
    17. Grabler, F., Agrawala, M., Li, W., Dontcheva, M., and Igarashi, T. 2009. Generating photo manipulation tutorials by demonstration. ACM Transactions on Graphics 28, 3, 66. Google ScholarDigital Library
    18. Grossman, T., Matejka, J., and Fitzmaurice, G. 2010. Chronicle: Capture, exploration, and playback of document workflow histories. In Proceedings of ACM Symposium on User Interface Software and Technology, 143–152. Google ScholarDigital Library
    19. Haase, C. S., and Meyer, G. W. 1992. Modeling pigmented materials for realistic image synthesis. ACM Transactions on Graphics 11, 4, 305–335. Google ScholarDigital Library
    20. Hu, S.-M., Xu, K., Ma, L.-Q., Liu, B., Jiang, B.-Y., and Wang, J. 2013. Inverse image editing: Recovering a semantic editing history from a before-and-after image pair. ACM Transactions on Graphics 32, 6, 194. Google ScholarDigital Library
    21. Hubbe, M. A., Pawlak, J. J., and Koukoulas, A. A. 2008. Paper’s appearance: A review. BioResources 3, 2, 627–665.Google ScholarCross Ref
    22. Karsch, K., Golparvar-Fard, M., and Forsyth, D. 2014. Constructaide: Analyzing and visualizing construction sites through photographs and building models. ACM Transactions on Graphics 33, 6, 176. Google ScholarDigital Library
    23. Konieczny, J., and Meyer, G. 2009. Airbrush simulation for artwork and computer modeling. In Proceedings of International Symposium on Non-Photorealistic Animation and Rendering, 61–69. Google ScholarDigital Library
    24. Kubelka, P., and Munk, F. 1931. An article on optics of paint layers. Zeitschrift für Technische Physik 12, 593–601.Google Scholar
    25. Kubelka, P. 1948. New contributions to the optics of intensely light-scattering materials. Part I. Journal of the Optical Society of America 38, 5, 448–448.Google ScholarCross Ref
    26. Kubelka, P. 1954. New contributions to the optics of intensely light-scattering materials. Part II: Nonhomogeneous layers. Journal of the Optical Society of America 44, 4, 330–334.Google ScholarCross Ref
    27. Lu, J., DiVerdi, S., Chen, W. A., Barnes, C., and Finkelstein, A. 2014. RealPigment: Paint compositing by example. In Proceedings of International Symposium on Non-Photorealistic Animation and Rendering, 21–30. Google ScholarDigital Library
    28. Matzen, K., and Snavely, N. 2014. Scene chronology. In Proceedings of European Conference on Computer Vision. 615–630.Google Scholar
    29. McCann, J., and Pollard, N. 2009. Local layering. ACM Transactions on Graphics 28, 3, 84. Google ScholarDigital Library
    30. McCann, J., and Pollard, N. 2012. Soft stacking. Computer Graphics Forum 31, 2, 469–478. Google ScholarDigital Library
    31. Nancel, M., and Cockburn, A. 2014. Causality: A conceptual model of interaction history. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 1777–1786. Google ScholarDigital Library
    32. Noris, G., Sýkora, D., Shamir, A., Coros, S., Whited, B., Simmons, M., Hornung, A., Gross, M., and Sumner, R. 2012. Smart scribbles for sketch segmentation. Computer Graphics Forum 31, 8, 2516–2527. Google ScholarDigital Library
    33. Novick, L. R., and Tversky, B. 1987. Cognitive constraints on ordering operations: The case of geometric analogies. Journal of Experimental Psychology: General 116, 1, 50.Google ScholarCross Ref
    34. Porter, T., and Duff, T. 1984. Compositing digital images. ACM SIGGRAPH Computer Graphics 18, 3, 253–259. Google ScholarDigital Library
    35. Richardt, C., Lopez-Moreno, J., Bousseau, A., Agrawala, M., and Drettakis, G. 2014. Vectorising bitmaps into semi-transparent gradient layers. Computer Graphics Forum (Proceedings of EGSR) 33, 4 (July), 11–19. Google ScholarDigital Library
    36. Rubinstein, M., Liu, C., Sand, P., Durand, F., and Freeman, W. T. 2011. Motion denoising with application to time-lapse photography. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 313–320. Google ScholarDigital Library
    37. Smith, A. R., and Blinn, J. F. 1996. Blue screen matting. In ACM SIGGRAPH Conference Proceedings, 259–268. Google ScholarDigital Library
    38. Su, S. L., Paris, S., and Durand, F. 2009. QuickSelect: History-based selection expansion. In Proceedings of Graphics Interface, 215–221. Google ScholarDigital Library
    39. Szeliski, R., Avidan, S., and Anandan, P. 2000. Layer extraction from multiple images containing reflections and transparency. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, vol. 1, 246–253.Google Scholar
    40. Tange, O. 2011. GNU Parallel: The command-line power tool. ;login: The USENIX Magazine 36, 1 (Feb), 42–47.Google Scholar
    41. Taylor, H. A., and Tversky, B. 1992. Descriptions and depictions of environments. Memory & Cognition 20, 5, 483–496.Google ScholarCross Ref
    42. Tomasi, C., and Manduchi, R. 1998. Bilateral filtering for gray and color images. In Proceedings of IEEE International Conference on Computer Vision, 839–846. Google ScholarDigital Library
    43. Tversky, B. 1999. What does drawing reveal about thinking? Visual and Spatial Reasoning in Design, 93–101.Google Scholar
    44. van Sommers, P. 1984. Drawing and cognition: Descriptive and experimental studies of graphic production processes. Cambridge University Press.Google Scholar
    45. VisTrails, I., 2009. Vistrails provenance explorer for maya. http://www.vistrails.com/maya.html.Google Scholar
    46. Xing, J., Chen, H.-T., and Wei, L.-Y. 2014. Autocomplete painting repetitions. ACM Transactions on Graphics 33, 6, 172. Google ScholarDigital Library
    47. Xu, S., Xu, Y., Kang, S. B., Salesin, D. H., Pan, Y., and Shum, H.-Y. 2006. Animating chinese paintings through stroke-based decomposition. ACM Transactions on Graphics 25, 2, 239–267. Google ScholarDigital Library
    48. Xu, L., Lu, C., Xu, Y., and Jia, J. 2011. Image smoothing via L0 gradient minimization. ACM Transactions on Graphics 30, 6, 174. Google ScholarDigital Library
    49. Zivkovic, Z., and van der Heijden, F. 2006. Efficient adaptive density estimation per image pixel for the task of background subtraction. Pattern Recognition Letters 27, 7, 773–780. Google ScholarDigital Library
    50. Zongker, D. E., Werner, D. M., Curless, B., and Salesin, D. H. 1999. Environment matting and compositing. In ACM SIGGRAPH Conference Proceedings, 205–214. Google ScholarDigital Library

ACM Digital Library Publication: