“A computational approach for obstruction-free photography” by Xue, Rubinstein, Liu and Freeman

  • ©Tianfan Xue, Michael Rubinstein, Ce Liu, and William T. Freeman

Conference:


Type:


Title:

    A computational approach for obstruction-free photography

Session/Category Title:   Image Processing


Presenter(s)/Author(s):


Moderator(s):



Abstract:


    We present a unified computational approach for taking photos through reflecting or occluding elements such as windows and fences. Rather than capturing a single image, we instruct the user to take a short image sequence while slightly moving the camera. Differences that often exist in the relative position of the background and the obstructing elements from the camera allow us to separate them based on their motions, and to recover the desired background scene as if the visual obstructions were not there. We show results on controlled experiments and many real and practical scenarios, including shooting through reflections, fences, and raindrop-covered windows.

References:


    1. Barnum, P. C., Narasimhan, S., and Kanade, T. 2010. Analysis of rain and snow in frequency space. International Journal of Computer Vision (IJCV) 86, 2-3, 256–274. Google ScholarDigital Library
    2. Be, E., Yeredor, A., and Member, S. 2008. Blind Separation of Superimposed Shifted Images Using Parameterized Joint Diagonalization. IEEE Transactions on Image Processing 17, 3, 340–353. Google ScholarDigital Library
    3. Bertalmio, M., Sapiro, G., Caselles, V., and Ballester, C. 2000. Image inpainting. In Computer Graphics and Interactive Techniques. Google ScholarDigital Library
    4. Bertalmio, M., Bertozzi, A. L., and Sapiro, G. 2001. Navier-stokes, fluid dynamics, and image and video inpainting. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
    5. Black, M. J., and Anandan, P. 1996. The robust estimation of multiple motions: Parametric and piecewise-smooth flow fields. Computer Vision and Image Understanding 63, 1, 75–104. Google ScholarDigital Library
    6. Canny, J. 1986. A computational approach to edge detection. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 6, 679–698. Google ScholarDigital Library
    7. Criminisi, A., Pérez, P., and Toyama, K. 2004. Region filling and object removal by exemplar-based image inpainting. IEEE Transactions on Image Processing 13, 9, 1200–1212. Google ScholarDigital Library
    8. Farid, H., and Adelson, E. H. 1999. Separating reflections and lighting using independent components analysis. IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
    9. Gai, K., Shi, Z., and Zhang, C. 2009. Blind separation of superimposed images with unknown motions. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
    10. Gai, K., Shi, Z., and Zhang, C. 2012. Blind separation of superimposed moving images using image statistics. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 34, 1, 19–32. Google ScholarDigital Library
    11. Garg, K., and Nayar, S. K. 2004. Detection and removal of rain from videos. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
    12. Guo, X., Cao, X., and Ma, Y. 2014. Robust Separation of Reflection from Multiple Images. IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Google ScholarDigital Library
    13. Hays, J., Leordeanu, M., Efros, A. A., and Liu, Y. 2006. Discovering texture regularity as a higher-order correspondence problem. In European Conference on Computer Vision (ECCV). Google ScholarDigital Library
    14. Jepson, A., and Black, M. J. 1993. Mixture models for optical flow computation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
    15. Jojic, N., and Frey, B. J. 2001. Learning flexible sprites in video layers. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
    16. Kong, N., Tai, Y.-W., and Shin, J. S. 2014. A physically-based approach to reflection separation: from physical modeling to constrained optimization. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 36, 2, 209–21. Google ScholarDigital Library
    17. Kopf, J., Langguth, F., Scharstein, D., Szeliski, R., Goesele, M., and Darmstadt, T. U. 2013. Image-Based Rendering in the Gradient Domain. ACM SIGGRAPH. Google ScholarDigital Library
    18. Levin, A., and Weiss, Y. 2007. User assisted separation of reflections from a single image using a sparsity prior. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 29, 9, 1647–1654. Google ScholarDigital Library
    19. Levin, A., Zomet, A., and Weiss, Y. 2002. Learning to perceive transparency from the statistics of natural scenes. Advances in Neural Information Processing Systems (NIPS).Google Scholar
    20. Levin, A., Zomet, A., and Weiss, Y. 2004. Separating reflections from a single image using local features. IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
    21. Li, Y., and Brown, M. S. 2013. Exploiting Reflection Change for Automatic Reflection Removal. IEEE International Conference on Computer Vision (ICCV). Google ScholarDigital Library
    22. Li, Y., and Brown, M. S. 2014. Single image layer separation using relative smoothness. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Google ScholarDigital Library
    23. Liu, C., and Sun, D. 2014. On bayesian adaptive video super resolution. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 36, 2, 346–360. Google ScholarDigital Library
    24. Liu, C., Yuen, J., Torralba, A., Sivic, J., and Freeman, W. T. 2008. Sift flow: Dense correspondence across different scenes. In European Conference on Computer Vision (ECCV). Google ScholarDigital Library
    25. Liu, S., Yuan, L., Tan, P., and Sun, J. 2014. Steadyflow: Spatially smooth optical flow for video stabilization. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Google ScholarDigital Library
    26. Luenberger, D. G. 1973. Introduction to linear and nonlinear programming, vol. 28. Addison-Wesley Reading, MA.Google Scholar
    27. Mu, Y., Liu, W., and Yan, S. 2012. Video de-fencing. IEEE Circuits and Systems Society.Google Scholar
    28. Newson, A., Almansa, A., Fradet, M., Gousseau, Y., Pérez, P., et al. 2014. Video inpainting of complex scenes. Journal on Imaging Sciences, Society for Industrial and Applied Mathematics.Google Scholar
    29. Park, M., Collins, R. T., and Liu, Y. 2008. Deformed lattice discovery via efficient mean-shift belief propagation. In European Conference on Computer Vision (ECCV). Google ScholarDigital Library
    30. Park, M., Brocklehurst, K., Collins, R. T., and Liu, Y. 2011. Image de-fencing revisited. In Asian Conference on Computer Vision (ACCV). Google ScholarDigital Library
    31. Sarel, B., and Irani, M. 2004. Separating transparent layers through layer information exchange. European Conference on Computer Vision (ECCV).Google Scholar
    32. Singh, M. 2003. Computing Layered Surface Representations : An Algorithm for Detecting and Separating Transparent Overlays. IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google ScholarCross Ref
    33. Sinha, S., Kopf, J., Goesele, M., Scharstein, D., and Szeliski, R. 2012. Image-based rendering for scenes with reflections. ACM SIGGRAPH. Google ScholarDigital Library
    34. Szeliski, R., Avidan, S., and Anandan, P. 2000. Layer extraction from multiple images containing reflections and transparency. IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
    35. Szeliski, R. 1990. Fast surface interpolation using hierarchical basis functions. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 12, 6, 513–528. Google ScholarDigital Library
    36. Tsin, Y., Kang, S. B., and Szeliski, R. 2006. Stereo matching with linear superposition of layers. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 28, 2, 290–301. Google ScholarDigital Library
    37. Weiss, Y., and Adelson, E. H. 1996. A unified mixture framework for motion segmentation: Incorporating spatial coherence and estimating the number of models. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE. Google ScholarDigital Library
    38. Yamashita, A., Matsui, A., and Kaneko, T. 2010. Fence removal from multi-focus images. In International Conference on Pattern Recognition (ICPR). Google ScholarDigital Library


ACM Digital Library Publication:



Overview Page: