“First-person hyper-lapse videos” by Kopf, Cohen and Szeliski

  • ©Johannes Kopf, Michael Cohen, and Richard Szeliski

Conference:


Type:


Title:

    First-person hyper-lapse videos

Session/Category Title: Video Applications

Presenter(s)/Author(s):


Moderator(s):


Abstract:


    We present a method for converting first-person videos, for example, captured with a helmet camera during activities such as rock climbing or bicycling, into hyper-lapse videos, i.e., time-lapse videos with a smoothly moving camera. At high speed-up rates, simple frame sub-sampling coupled with existing video stabilization methods does not work, because the erratic camera shake present in first-person videos is amplified by the speed-up. Our algorithm first reconstructs the 3D input camera path as well as dense, per-frame proxy geometries. We then optimize a novel camera path for the output video that passes near the input cameras while ensuring that the virtual camera looks in directions that can be rendered well from the input. Finally, we generate the novel smoothed, time-lapse video by rendering, stitching, and blending appropriately selected source frames for each output frame. We present a number of results for challenging videos that cannot be processed using traditional techniques.

References:


    1. Agarwala, A., Dontcheva, M., Agrawala, M., Drucker, S., Colburn, A., Curless, B., Salesin, D., and Cohen, M. 2004. Interactive digital photomontage. ACM Trans. Graph. 23, 3, 294–302. Google ScholarDigital Library
    2. Beardsley, P., Torr, P., and Zisserman, A. 1996. 3D model acquisition from extended image sequences. In Fourth European Conference on Computer Vision (ECCV’96), Springer-Verlag, vol. 2, 683–695. Google ScholarDigital Library
    3. Buehler, C., Bosse, M., McMillan, L., Gortler, S., and Cohen, M. 2001. Unstructured lumigraph rendering. Proceedings of SIGGRAPH ’01, 425–432. Google ScholarDigital Library
    4. Chen, F., Cooper, M., and Adcock, J. 2007. Video summarization preserving dynamic content. In Proceedings of the International Workshop on TRECVID Video Summarization, ACM, New York, NY, USA, TVS ’07, 40–44. Google ScholarDigital Library
    5. Crandall, D., Owens, A., Snavely, N., and Huttenlocher, D. 2013. SfM with MRFs: Discrete-continuous optimization for large-scale reconstruction. IEEE Transactions on Pattern Analysis and Machine Intelligence 35, 12, 2841–2853. Google ScholarDigital Library
    6. Detyniecki, M., and Marsala, C. 2008. Adaptive acceleration and shot stacking for video rushes summarization. In Proceedings of the 2Nd ACM TRECVid Video Summarization Workshop, ACM, New York, NY, USA, TVS ’08, 109–113. Google ScholarDigital Library
    7. Furukawa, Y., and Ponce, J. 2010. Accurate, dense, and robust multi-view stereopsis. IEEE Transactions on Pattern Analysis and Machine Intelligence 32, 8, 1362–1376. Google ScholarDigital Library
    8. Goldstein, A., and Fattal, R. 2012. Video stabilization using epipolar geometry. ACM Trans. Graph. 32, 5. Google ScholarDigital Library
    9. Grundmann, M., Kwatra, V., and Essa, I. 2011. Auto-directed video stabilization with robust l1 optimal camera paths. IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Google ScholarDigital Library
    10. Hastie, T., Tibshirani, R., Friedman, J., and Franklin, J. 2005. The elements of statistical learning: data mining, inference and prediction. The Mathematical Intelligencer 27, 2, 83–85.Google ScholarCross Ref
    11. Horn, B. K. P. 1987. Closed-form solution of absolute orientation using unit quaternions. Journal of the Optical Society of America A 4, 4, 629–642.Google ScholarCross Ref
    12. Klein, G., and Murray, D. 2009. Parallel tracking and mapping on a camera phone. International Symposium on Mixed and Augmented Reality (ISMAR 2009). Google ScholarDigital Library
    13. Kolmogorov, V., and Zabih, R. 2004. What energy functions can be minimized via graph cuts. IEEE Transactions on Pattern Analysis and Machine Intelligence 26, 65–81. Google ScholarDigital Library
    14. Liu, F., Gleicher, M., Jin, H., and Agarwala, A. 2009. Content-preserving warps for 3d video stabilization. ACM Trans. Graph. 28, 3, article no. 44. Google ScholarDigital Library
    15. Liu, F., Gleicher, M., Wang, J., Jin, H., and Agarwala, A. 2011. Subspace video stabilization. ACM Trans. Graph. 30, 1 (Feb.), 4:1–4:10. Google ScholarDigital Library
    16. Liu, S., Yuan, L., Tan, P., and Sun, J. 2013. Bundled camera paths for video stabilization. ACM Transactions on Graphics (Proc. SIGGRAPH 2013) 32, 4, article no. 78. Google ScholarDigital Library
    17. Lombaert, H., Sun, Y., Grady, L., and Xu, C. 2005. A multilevel banded graph cuts method for fast image segmentation. IEEE International Conference on Computer Vision (ICCV’05), 259–265. Google ScholarDigital Library
    18. Matsushita, Y., Ofek, E., Ge, W., Tang, X., and Shum, H.-Y. 2006. Full-frame video stabilization with motion inpainting. IEEE Transactions on Pattern Analysis and Machine Intelligence 28, 7, 1150–1163. Google ScholarDigital Library
    19. Money, A. G., and Agius, H. 2008. Video summarisation: A conceptual framework and survey of the state of the art. Journal of Visual Communication and Image Representation 19, 2, 121–143. Google ScholarDigital Library
    20. Nocedal, J., and Wright, S. J. 2000. Numerical Optimization. Springer.Google Scholar
    21. Scaramuzza, D., Martinelli, A., and Siegwart, R. 2006. A toolbox for easily calibrating omnidirectional cameras. Proceedings of IEEE International Conference on Intelligent Robots and Systems (IROS 2006), 5695–5701.Google Scholar
    22. Schwarz, G. 1978. Estimating the Dimension of a Model. The Annals of Statistics 6, 2, 461–464.Google ScholarCross Ref
    23. Shewchuk, J. R. 1994. An introduction to the conjugate gradient method without the agonizing pain. Tech. rep., Carnegie Mellon University. Google ScholarDigital Library
    24. Snavely, N., Seitz, S. M., and Szeliski, R. 2006. Photo tourism: Exploring photo collections in 3d. ACM Transactions on Graphics (Proc. SIGGRAPH 2006) 25, 3, 835–846. Google ScholarDigital Library
    25. Snavely, N., Garg, R., Seitz, S. M., and Szeliski, R. 2008. Finding paths through the world’s photos. ACM Transactions on Graphics (Proc. SIGGRAPH 2008) 27, 3, article no. 15. Google ScholarDigital Library
    26. Szeliski, R., Uyttendaele, M., and Steedly, D. 2011. Fast poisson blending using multi-splines. International Conference on Computational Photography (ICCP 11).Google Scholar
    27. Taubin, G., Zhang, T., and Golub, G. 1996. Optimal surface smoothing as filter design. In Fourth European Conference on Computer Vision (ECCV’96), Springer-Verlag, vol. 1, 283–292. Google ScholarDigital Library
    28. Wang, W., Pottmann, H., and Liu, Y. 2006. Fitting b-spline curves to point clouds by curvature-based squared distance minimization. ACM Trans. Graph. 25, 2, 214–238. Google ScholarDigital Library
    29. Wu, C. 2013. Towards linear-time incremental structure from motion. In International Conference on 3D Vision (3DV). Google ScholarDigital Library

ACM Digital Library Publication:


Overview Page: