“A computational approach for obstruction-free photography” by Xue, Rubinstein, Liu and Freeman
Conference:
Type(s):
Title:
- A computational approach for obstruction-free photography
Session/Category Title: Image Processing
Presenter(s)/Author(s):
Moderator(s):
Abstract:
We present a unified computational approach for taking photos through reflecting or occluding elements such as windows and fences. Rather than capturing a single image, we instruct the user to take a short image sequence while slightly moving the camera. Differences that often exist in the relative position of the background and the obstructing elements from the camera allow us to separate them based on their motions, and to recover the desired background scene as if the visual obstructions were not there. We show results on controlled experiments and many real and practical scenarios, including shooting through reflections, fences, and raindrop-covered windows.
References:
1. Barnum, P. C., Narasimhan, S., and Kanade, T. 2010. Analysis of rain and snow in frequency space. International Journal of Computer Vision (IJCV) 86, 2-3, 256–274. Google ScholarDigital Library
2. Be, E., Yeredor, A., and Member, S. 2008. Blind Separation of Superimposed Shifted Images Using Parameterized Joint Diagonalization. IEEE Transactions on Image Processing 17, 3, 340–353. Google ScholarDigital Library
3. Bertalmio, M., Sapiro, G., Caselles, V., and Ballester, C. 2000. Image inpainting. In Computer Graphics and Interactive Techniques. Google ScholarDigital Library
4. Bertalmio, M., Bertozzi, A. L., and Sapiro, G. 2001. Navier-stokes, fluid dynamics, and image and video inpainting. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
5. Black, M. J., and Anandan, P. 1996. The robust estimation of multiple motions: Parametric and piecewise-smooth flow fields. Computer Vision and Image Understanding 63, 1, 75–104. Google ScholarDigital Library
6. Canny, J. 1986. A computational approach to edge detection. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 6, 679–698. Google ScholarDigital Library
7. Criminisi, A., Pérez, P., and Toyama, K. 2004. Region filling and object removal by exemplar-based image inpainting. IEEE Transactions on Image Processing 13, 9, 1200–1212. Google ScholarDigital Library
8. Farid, H., and Adelson, E. H. 1999. Separating reflections and lighting using independent components analysis. IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
9. Gai, K., Shi, Z., and Zhang, C. 2009. Blind separation of superimposed images with unknown motions. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
10. Gai, K., Shi, Z., and Zhang, C. 2012. Blind separation of superimposed moving images using image statistics. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 34, 1, 19–32. Google ScholarDigital Library
11. Garg, K., and Nayar, S. K. 2004. Detection and removal of rain from videos. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
12. Guo, X., Cao, X., and Ma, Y. 2014. Robust Separation of Reflection from Multiple Images. IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Google ScholarDigital Library
13. Hays, J., Leordeanu, M., Efros, A. A., and Liu, Y. 2006. Discovering texture regularity as a higher-order correspondence problem. In European Conference on Computer Vision (ECCV). Google ScholarDigital Library
14. Jepson, A., and Black, M. J. 1993. Mixture models for optical flow computation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
15. Jojic, N., and Frey, B. J. 2001. Learning flexible sprites in video layers. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
16. Kong, N., Tai, Y.-W., and Shin, J. S. 2014. A physically-based approach to reflection separation: from physical modeling to constrained optimization. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 36, 2, 209–21. Google ScholarDigital Library
17. Kopf, J., Langguth, F., Scharstein, D., Szeliski, R., Goesele, M., and Darmstadt, T. U. 2013. Image-Based Rendering in the Gradient Domain. ACM SIGGRAPH. Google ScholarDigital Library
18. Levin, A., and Weiss, Y. 2007. User assisted separation of reflections from a single image using a sparsity prior. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 29, 9, 1647–1654. Google ScholarDigital Library
19. Levin, A., Zomet, A., and Weiss, Y. 2002. Learning to perceive transparency from the statistics of natural scenes. Advances in Neural Information Processing Systems (NIPS).Google Scholar
20. Levin, A., Zomet, A., and Weiss, Y. 2004. Separating reflections from a single image using local features. IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
21. Li, Y., and Brown, M. S. 2013. Exploiting Reflection Change for Automatic Reflection Removal. IEEE International Conference on Computer Vision (ICCV). Google ScholarDigital Library
22. Li, Y., and Brown, M. S. 2014. Single image layer separation using relative smoothness. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Google ScholarDigital Library
23. Liu, C., and Sun, D. 2014. On bayesian adaptive video super resolution. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 36, 2, 346–360. Google ScholarDigital Library
24. Liu, C., Yuen, J., Torralba, A., Sivic, J., and Freeman, W. T. 2008. Sift flow: Dense correspondence across different scenes. In European Conference on Computer Vision (ECCV). Google ScholarDigital Library
25. Liu, S., Yuan, L., Tan, P., and Sun, J. 2014. Steadyflow: Spatially smooth optical flow for video stabilization. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Google ScholarDigital Library
26. Luenberger, D. G. 1973. Introduction to linear and nonlinear programming, vol. 28. Addison-Wesley Reading, MA.Google Scholar
27. Mu, Y., Liu, W., and Yan, S. 2012. Video de-fencing. IEEE Circuits and Systems Society.Google Scholar
28. Newson, A., Almansa, A., Fradet, M., Gousseau, Y., Pérez, P., et al. 2014. Video inpainting of complex scenes. Journal on Imaging Sciences, Society for Industrial and Applied Mathematics.Google Scholar
29. Park, M., Collins, R. T., and Liu, Y. 2008. Deformed lattice discovery via efficient mean-shift belief propagation. In European Conference on Computer Vision (ECCV). Google ScholarDigital Library
30. Park, M., Brocklehurst, K., Collins, R. T., and Liu, Y. 2011. Image de-fencing revisited. In Asian Conference on Computer Vision (ACCV). Google ScholarDigital Library
31. Sarel, B., and Irani, M. 2004. Separating transparent layers through layer information exchange. European Conference on Computer Vision (ECCV).Google Scholar
32. Singh, M. 2003. Computing Layered Surface Representations : An Algorithm for Detecting and Separating Transparent Overlays. IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google ScholarCross Ref
33. Sinha, S., Kopf, J., Goesele, M., Scharstein, D., and Szeliski, R. 2012. Image-based rendering for scenes with reflections. ACM SIGGRAPH. Google ScholarDigital Library
34. Szeliski, R., Avidan, S., and Anandan, P. 2000. Layer extraction from multiple images containing reflections and transparency. IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
35. Szeliski, R. 1990. Fast surface interpolation using hierarchical basis functions. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 12, 6, 513–528. Google ScholarDigital Library
36. Tsin, Y., Kang, S. B., and Szeliski, R. 2006. Stereo matching with linear superposition of layers. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 28, 2, 290–301. Google ScholarDigital Library
37. Weiss, Y., and Adelson, E. H. 1996. A unified mixture framework for motion segmentation: Incorporating spatial coherence and estimating the number of models. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE. Google ScholarDigital Library
38. Yamashita, A., Matsui, A., and Kaneko, T. 2010. Fence removal from multi-focus images. In International Conference on Pattern Recognition (ICPR). Google ScholarDigital Library
