“Factored time-lapse video” by Sunkavalli, Matusik, Pfister and Rusinkiewicz

We describe a method for converting time-lapse photography captured with outdoor cameras into Factored Time-Lapse Video (FTLV): a video in which time appears to move faster (i.e., lapsing) and where data at each pixel has been factored into shadow, illumination, and reflectance components. The factorization allows a user to easily relight the scene, recover a portion of the scene geometry (normals), and to perform advanced image editing operations. Our method is easy to implement, robust, and provides a compact representation with good reconstruction characteristics. We show results using several publicly available time-lapse sequences.

1. Agarwala, A., Zheng, K. C., Pal, C., Agrawala, M., Cohen, M., Curless, B., Salesin, D., and Szeliski, R. 2005. Panoramic video textures. ACM Trans. on Graph. 24, 3, 821–827. Google ScholarDigital Library
2. Barrow, H., and Tenenbaum, J. 1978. Recovering intrinsic scene characteristics from images. Academic Press, 3–26.Google Scholar
3. Bhat, K., Seitz, S., Hodgins, J. K., and Khosla, P. 2004. Flow-based video synthesis and editing. ACM Trans. on Graph. 23, 3, 360–363. Google ScholarDigital Library
4. Bregler, C., Covell, M., and Slaney, M. 1997. Video rewrite: driving visual speech with audio. In Proc. of ACM SIGGRAPH, ACM Press, New York, NY, USA, 353–360. Google ScholarDigital Library
5. Chuang, Y.-Y, Agarwala, A., Curless, B., Salesin, D. H., and Szeliski, R. 2002. Video matting of complex scenes. ACM Trans. on Graph, 21, 3 (July), 243–248. Google ScholarDigital Library
6. Chuang, Y.-Y., Goldman, D. B., Curless, B., Salesin, D. H., and Szeliski, R. 2003. Shadow matting and compositing. ACM Trans. on Graph. 22, 3, 494–500. Google ScholarDigital Library
7. Debevec, P., Tchou, C., Gardner, A., Hawkins, T., Poullis, C., Stumpfel, J., Jones, A., Yun, N., Einarsson, P., Lundgren, T., Fajardo, M., and Martinez, P., 2004. Estimating Surface Reflectance Properties of a Complex Scene under Captured Natural Illumination. USC ICT Technical Report ICT-TR-06.2004.Google Scholar
8. Gu, J., Tu, C.-I., Ramamoorthi, R., Belhumeur, P., Matusik, W., and Nayar, S. 2006. Time-varying surface appearance: acquisition, modeling and rendering. ACM Trans. on Graph. 25, 3, 762–771. Google ScholarDigital Library
9. Hertzmann, A., and Seitz, S. M. 2005. Example-based photometric stereo: Shape reconstruction with general, varying brdfs. IEEE Trans. on PAMI 27, 8, 1254–1264. Google ScholarDigital Library
10. Koppal, S. J., and Narasimhan, S. G. 2006. Clustering appearance for scene analysis. In Proc. of CVPR, vol. 2, 1323–1330. Google ScholarDigital Library
11. Lawrence, J., Ben-Artzi, A., DeCoro, C., Matusik, W., Pfister, H., Ramamoorthi, R., and Rusinkiewicz, S. 2006. Inverse shade trees for non-parametric material representation and editing. ACM Trans. on Graph. 25, 3, 735–745. Google ScholarDigital Library
12. Li, Y., Sun J., and Shum, H.-Y. 2005. Video object cut and paste. ACM Trans. on Graph. 24, 3, 595–600. Google ScholarDigital Library
13. Litwinowicz, P. 1997. Processing images and video for an impressionist effect. In Proc. of ACM SIGGRAPH, ACM Press, New York, NY, USA, 407–414. Google ScholarDigital Library
14. Liu, C., Torralba, A., Freeman, W. T., Durand, F., and Adelson, E. H. 2005. Motion magnification. ACM Trans. on Graph. 24, 3, 519–526. Google ScholarDigital Library
15. Matsushita, Y., Nishino, K., Ikeuchi, K., and Sakauchi, M. 2004. Illumination normalization with time-dependent intrinsic images for video surveillance. IEEE Trans. on PAMI 26, 10, 1336–1347. Google ScholarDigital Library
16. Matusik, W., Loper, M., and Pfister, H. 2004. Progressively-refined reflectance functions from natural illumination. In Rendering Techniques, Eurographics Association, A. Keller and H. W. Jensen, Eds., 299–308. Google ScholarCross Ref
17. McGuire, M., Matusik, W., Pfister, H., Hughes, J. F., and Durand, F. 2005. Defocus video matting. ACM Trans. on Graph. 24, 3, 567–576. Google ScholarDigital Library
18. Nayar, S. K., Krishnan, G., Grossberg, M. D., and Raskar, R. 2006. Fast separation of direct and global components of a scene using high frequency illumination. ACM Trans. on Graph. 25, 3, 935–944. Google ScholarDigital Library
19. Nimeroff, J. S., Simoncelli, E., and Dorsey, J. 1994. Efficient Re-rendering of Naturally Illuminated Environments.Google Scholar
20. In Fifth Eurographics Workshop on Rendering, Springer-Verlag, Darmstadt, Germany, 359–373.Google Scholar
21. Rusinkiewicz, S., Burns, M., and DeCarlo, D. 2006. Exaggerated shading for depicting shape and detail. ACM Trans, on Graph. 25, 3, 1199–1205. Google ScholarDigital Library
22. Schödl, A., Szeliski, R., Salesin, D., and Essa, I. 2000. Video textures. In Proc. of ACM SIGGRAPH, ACM Press, New York, NY, USA, 489–498. Google ScholarDigital Library
23. Seitz, S. M., Matsushita, Y., and Kutulakos, K. N. 2005. A theory of inverse light transport. In Proc. of ICCV, II: 1440–1447. Google ScholarDigital Library
24. Tomasi, C., and Manduchi, R. 1998. Bilateral filtering for gray and color images. In Proc. of ICCV, 839–846. Google ScholarDigital Library
25. Wang, J., Xu, Y., Shum, H.-Y., and Cohen, M. F. 2004. Video tooning. ACM Trans. on Graph. 23, 3, 574–583. Google ScholarDigital Library
26. Wang, J., Bhat, P., Colburn, R. A., Agrawala, M., and Cohen, M. F. 2005. Interactive video cutout. ACM Trans. on Graph. 24, 3, 585–594. Google ScholarDigital Library
27. Weiss, Y. 2001. Deriving intrinsic images from image sequences. In Proc. of ICCV, II: 68–75.Google Scholar
28. Winnemöller, H., Olsen, S., and Gooch, B. 2006. Realtime video abstraction. ACM Trans. on Graph. 25, 3, 1221–1226. Google ScholarDigital Library
29. Yu, Y., and Malik, J. 1998. Recovering photometric properties of architectural scenes from photographs. In Proc. of ACM SIGGRAPH, ACM Press, New York, NY, USA, 207–217. Google ScholarDigital Library