“Patch-based high dynamic range video”
Conference:
Type(s):
Title:
- Patch-based high dynamic range video
Session/Category Title: HDR & IBR
Presenter(s)/Author(s):
Abstract:
Despite significant progress in high dynamic range (HDR) imaging over the years, it is still difficult to capture high-quality HDR video with a conventional, off-the-shelf camera. The most practical way to do this is to capture alternating exposures for every LDR frame and then use an alignment method based on optical flow to register the exposures together. However, this results in objectionable artifacts whenever there is complex motion and optical flow fails. To address this problem, we propose a new approach for HDR reconstruction from alternating exposure video sequences that combines the advantages of optical flow and recently introduced patch-based synthesis for HDR images. We use patch-based synthesis to enforce similarity between adjacent frames, increasing temporal continuity. To synthesize visually plausible solutions, we enforce constraints from motion estimation coupled with a search window map that guides the patch-based synthesis. This results in a novel reconstruction algorithm that can produce high-quality HDR videos with a standard camera. Furthermore, our method is able to synthesize plausible texture and motion in fast-moving regions, where either patch-based synthesis or optical flow alone would exhibit artifacts. We present results of our reconstructed HDR video sequences that are superior to those produced by current approaches.
References:
1. Adams, A., Talvala, E.-V., Park, S. H., Jacobs, D. E., Ajdin, B., Gelfand, N., Dolson, J., Vaquero, D., Baek, J., Tico, M., Lensch, H. P. A., Matusik, W., Pulli, K., Horowitz, M., and Levoy, M. 2010. The frankencamera: an experimental platform for computational photography. ACM Trans. Graph. 29, 4 (July), 29:1–29:12.
2. Barnes, C., Shechtman, E., Finkelstein, A., and Goldman, D. B. 2009. Patchmatch: a randomized correspondence algorithm for structural image editing. ACM Trans. Graph. 28 (July), 24:1–24:11.
3. Brajovic, V., and Kanade, T. 1996. A sorting image sensor: an example of massively parallel intensity-to-time processing for low-latency computational sensors. In Proceedings of ICRA, 1996, vol. 2, 1638–1643.
4. Brox, T., and Malik, J. 2011. Large displacement optical flow: Descriptor matching in variational motion estimation. IEEE Trans. Pattern Anal. Mach. Intell. 33, 3 (Mar.), 500–513.
5. Cole, A., and Safai, M., 2013. Soviet Montage Productions. http://www.sovietmontage.com/.
6. Debevec, P. E., and Malik, J. 1997. Recovering high dynamic range radiance maps from photographs. In Proceedings of ACM SIGGRAPH 1997, 369–378.
7. Ginger HDR, 2013. A commercial HDR merging application. http://www.19lights.com/.
8. Jahne, B., Geissler, P., and Haussecker, H., Eds. 1999. Handbook of Computer Vision and Applications with Cdrom, 1st ed., vol. 2. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA.
9. Kang, S. B., Uyttendaele, M., Winder, S., and Szeliski, R. 2003. High dynamic range video. ACM Trans. Graph. 22, 3 (July), 319–325.
10. Kronander, J., Gustavson, S., Bonnet, G., and Unger, J. 2013. Unified HDR reconstruction from raw CFA data. IEEE International Conference on Computational Photography (ICCP).
11. Liu, C. 2009. Beyond Pixels: Exploring New Representations and Applications for Motion Analysis. Doctoral thesis, Massachusetts Institute of Technology.
12. Lowe, D. G. 2004. Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vision 60, 2 (Nov.), 91–110.
13. Magic Lantern, 2013. Canon DSLR camera firmware. http://www.magiclantern.fm/.
14. Mangiat, S., and Gibson, J. 2010. High dynamic range video with ghost removal. In Proc. SPIE 7798, no. 779812, 1–8.
15. Mangiat, S., and Gibson, J. 2011. Spatially adaptive filtering for registration artifact removal in HDR video. In ICIP 2011, 1317–1320.
16. Mangiat, S. 2012. High Dynamic Range and 3D Video Communications for Handheld Devices. Doctoral thesis, University of California, Santa Barbara.
17. Mann, S., and Picard, R. W. 1995. On being ‘undigital’ with digital cameras: Extending dynamic range by combining differently exposed pictures. In Proc. of Society for Imaging Science and Technology, 442–448.
18. McGuire, M., Matusik, W., Pfister, H., Chen, B., Hughes, J., and Nayar, S. 2007. Optical splitting trees for high-precision monocular imaging. IEEE Computer Graphics and Applications 27, 2 (march-april), 32–42.
19. Nayar, S., and Branzoi, V. 2003. Adaptive dynamic range imaging: optical control of pixel exposures over space and time. In Proceedings of ICCV 2003, 1168–1175.
20. Nayar, S., and Mitsunaga, T. 2000. High dynamic range imaging: spatially varying pixel exposures. In CVPR 2000, 472–479.
21. Portz, T., Zhang, L., and Jiang, H. 2013. Adaptive dynamic range imaging: optical control of pixel exposures over space and time. In Proceedings of ICCP 2013.
22. Reinhard, E., Stark, M., Shirley, P., and Ferwerda, J. 2002. Photographic tone reproduction for digital images. ACM Trans. Graph. 21, 3 (July), 267–276.
23. Reinhard, E., Heidrich, W., Debevec, P., Pattanaik, S., Ward, G., and Myszkowski, K. 2010. High Dynamic Range Imaging: Acquisition, Display, and Image-Based Lighting, second ed. Morgan Kaufmann.
24. Seger, U., Apel, U., and Höfflinger, B. 1999. HDRC-Imagers for natural visual perception. In Handbook of Computer Vision and Application, B. Jähne, H. Haußecker, and P. Geißler, Eds., vol. 1. Academic Press, 223–235.
25. Sen, P., Kalantari, N. K., Yaesoubi, M., Darabi, S., Goldman, D. B., and Shechtman, E. 2012. Robust patch-based HDR reconstruction of dynamic scenes. ACM Trans. Graph. 31, 6 (Nov.), 203:1–203:11.
26. Shechtman, E., Rav-Acha, A., Irani, M., and Seitz, S. 2010. Regenerative morphing. In CVPR 2010, 615–622.
27. Simakov, D., Caspi, Y., Shechtman, E., and Irani, M. 2008. Summarizing visual data using bidirectional similarity. In CVPR 2008, 1–8.
28. SpheronVR, 2013. http://www.spheron.com/.
29. Tocci, M. D., Kiser, C., Tocci, N., and Sen, P. 2011. A versatile HDR video production system. ACM Trans. Graph. 30, 4 (July), 41:1–41:10.
30. Unger, J., and Gustavson, S. 2007. High-dynamic-range video for photometric measurement of illumination. SPIE, vol. 6501, 65010E.
31. Zimmer, H., Bruhn, A., and Weickert, J. 2011. Freehand HDR imaging of moving scenes with simultaneous resolution enhancement. Computer Graphics Forum 30, 2 (Apr.), 405–414.
