“Factored time-lapse video” by Sunkavalli, Matusik, Pfister and Rusinkiewicz

  • ©Kalyan Sunkavalli, Wojciech Matusik, Hanspeter Pfister, and Szymon Rusinkiewicz




    Factored time-lapse video



    We describe a method for converting time-lapse photography captured with outdoor cameras into Factored Time-Lapse Video (FTLV): a video in which time appears to move faster (i.e., lapsing) and where data at each pixel has been factored into shadow, illumination, and reflectance components. The factorization allows a user to easily relight the scene, recover a portion of the scene geometry (normals), and to perform advanced image editing operations. Our method is easy to implement, robust, and provides a compact representation with good reconstruction characteristics. We show results using several publicly available time-lapse sequences.


    1. Agarwala, A., Zheng, K. C., Pal, C., Agrawala, M., Cohen, M., Curless, B., Salesin, D., and Szeliski, R. 2005. Panoramic video textures. ACM Trans. on Graph. 24, 3, 821–827. Google ScholarDigital Library
    2. Barrow, H., and Tenenbaum, J. 1978. Recovering intrinsic scene characteristics from images. Academic Press, 3–26.Google Scholar
    3. Bhat, K., Seitz, S., Hodgins, J. K., and Khosla, P. 2004. Flow-based video synthesis and editing. ACM Trans. on Graph. 23, 3, 360–363. Google ScholarDigital Library
    4. Bregler, C., Covell, M., and Slaney, M. 1997. Video rewrite: driving visual speech with audio. In Proc. of ACM SIGGRAPH, ACM Press, New York, NY, USA, 353–360. Google ScholarDigital Library
    5. Chuang, Y.-Y, Agarwala, A., Curless, B., Salesin, D. H., and Szeliski, R. 2002. Video matting of complex scenes. ACM Trans. on Graph, 21, 3 (July), 243–248. Google ScholarDigital Library
    6. Chuang, Y.-Y., Goldman, D. B., Curless, B., Salesin, D. H., and Szeliski, R. 2003. Shadow matting and compositing. ACM Trans. on Graph. 22, 3, 494–500. Google ScholarDigital Library
    7. Debevec, P., Tchou, C., Gardner, A., Hawkins, T., Poullis, C., Stumpfel, J., Jones, A., Yun, N., Einarsson, P., Lundgren, T., Fajardo, M., and Martinez, P., 2004. Estimating Surface Reflectance Properties of a Complex Scene under Captured Natural Illumination. USC ICT Technical Report ICT-TR-06.2004.Google Scholar
    8. Gu, J., Tu, C.-I., Ramamoorthi, R., Belhumeur, P., Matusik, W., and Nayar, S. 2006. Time-varying surface appearance: acquisition, modeling and rendering. ACM Trans. on Graph. 25, 3, 762–771. Google ScholarDigital Library
    9. Hertzmann, A., and Seitz, S. M. 2005. Example-based photometric stereo: Shape reconstruction with general, varying brdfs. IEEE Trans. on PAMI 27, 8, 1254–1264. Google ScholarDigital Library
    10. Koppal, S. J., and Narasimhan, S. G. 2006. Clustering appearance for scene analysis. In Proc. of CVPR, vol. 2, 1323–1330. Google ScholarDigital Library
    11. Lawrence, J., Ben-Artzi, A., DeCoro, C., Matusik, W., Pfister, H., Ramamoorthi, R., and Rusinkiewicz, S. 2006. Inverse shade trees for non-parametric material representation and editing. ACM Trans. on Graph. 25, 3, 735–745. Google ScholarDigital Library
    12. Li, Y., Sun J., and Shum, H.-Y. 2005. Video object cut and paste. ACM Trans. on Graph. 24, 3, 595–600. Google ScholarDigital Library
    13. Litwinowicz, P. 1997. Processing images and video for an impressionist effect. In Proc. of ACM SIGGRAPH, ACM Press, New York, NY, USA, 407–414. Google ScholarDigital Library
    14. Liu, C., Torralba, A., Freeman, W. T., Durand, F., and Adelson, E. H. 2005. Motion magnification. ACM Trans. on Graph. 24, 3, 519–526. Google ScholarDigital Library
    15. Matsushita, Y., Nishino, K., Ikeuchi, K., and Sakauchi, M. 2004. Illumination normalization with time-dependent intrinsic images for video surveillance. IEEE Trans. on PAMI 26, 10, 1336–1347. Google ScholarDigital Library
    16. Matusik, W., Loper, M., and Pfister, H. 2004. Progressively-refined reflectance functions from natural illumination. In Rendering Techniques, Eurographics Association, A. Keller and H. W. Jensen, Eds., 299–308. Google ScholarCross Ref
    17. McGuire, M., Matusik, W., Pfister, H., Hughes, J. F., and Durand, F. 2005. Defocus video matting. ACM Trans. on Graph. 24, 3, 567–576. Google ScholarDigital Library
    18. Nayar, S. K., Krishnan, G., Grossberg, M. D., and Raskar, R. 2006. Fast separation of direct and global components of a scene using high frequency illumination. ACM Trans. on Graph. 25, 3, 935–944. Google ScholarDigital Library
    19. Nimeroff, J. S., Simoncelli, E., and Dorsey, J. 1994. Efficient Re-rendering of Naturally Illuminated Environments.Google Scholar
    20. In Fifth Eurographics Workshop on Rendering, Springer-Verlag, Darmstadt, Germany, 359–373.Google Scholar
    21. Rusinkiewicz, S., Burns, M., and DeCarlo, D. 2006. Exaggerated shading for depicting shape and detail. ACM Trans, on Graph. 25, 3, 1199–1205. Google ScholarDigital Library
    22. Schödl, A., Szeliski, R., Salesin, D., and Essa, I. 2000. Video textures. In Proc. of ACM SIGGRAPH, ACM Press, New York, NY, USA, 489–498. Google ScholarDigital Library
    23. Seitz, S. M., Matsushita, Y., and Kutulakos, K. N. 2005. A theory of inverse light transport. In Proc. of ICCV, II: 1440–1447. Google ScholarDigital Library
    24. Tomasi, C., and Manduchi, R. 1998. Bilateral filtering for gray and color images. In Proc. of ICCV, 839–846. Google ScholarDigital Library
    25. Wang, J., Xu, Y., Shum, H.-Y., and Cohen, M. F. 2004. Video tooning. ACM Trans. on Graph. 23, 3, 574–583. Google ScholarDigital Library
    26. Wang, J., Bhat, P., Colburn, R. A., Agrawala, M., and Cohen, M. F. 2005. Interactive video cutout. ACM Trans. on Graph. 24, 3, 585–594. Google ScholarDigital Library
    27. Weiss, Y. 2001. Deriving intrinsic images from image sequences. In Proc. of ICCV, II: 68–75.Google Scholar
    28. Winnemöller, H., Olsen, S., and Gooch, B. 2006. Realtime video abstraction. ACM Trans. on Graph. 25, 3, 1221–1226. Google ScholarDigital Library
    29. Yu, Y., and Malik, J. 1998. Recovering photometric properties of architectural scenes from photographs. In Proc. of ACM SIGGRAPH, ACM Press, New York, NY, USA, 207–217. Google ScholarDigital Library

ACM Digital Library Publication: