“4D imaging through spray-on optics” by Iseringhausen, Goldlücke, Pesheva, Iliev, Wender, et al. …

  • ©Julian Iseringhausen, Bastian Goldlücke, Nina Pesheva, Stanimir Iliev, Alexander Wender, Martin Fuchs, and Matthias B. Hullin

Conference:


Type:


Title:

    4D imaging through spray-on optics

Session/Category Title: Imaginative Imaging

Presenter(s)/Author(s):


Moderator(s):


Abstract:


    Light fields are a powerful concept in computational imaging and a mainstay in image-based rendering; however, so far their acquisition required either carefully designed and calibrated optical systems (micro-lens arrays), or multi-camera/multi-shot settings. Here, we show that fully calibrated light field data can be obtained from a single ordinary photograph taken through a partially wetted window. Each drop of water produces a distorted view on the scene, and the challenge of recovering the unknown mapping from pixel coordinates to refracted rays in space is a severely underconstrained problem. The key idea behind our solution is to combine ray tracing and low-level image analysis techniques (extraction of 2D drop contours and locations of scene features seen through drops) with state-of-the-art drop shape simulation and an iterative refinement scheme to enforce photo-consistency across features that are seen in multiple views. This novel approach not only recovers a dense pixel-to-ray mapping, but also the refractive geometry through which the scene is observed, to high accuracy. We therefore anticipate that our inherently self-calibrating scheme might also find applications in other fields, for instance in materials science where the wetting properties of liquids on surfaces are investigated.

References:


    1. Adamson, A. and A. Gast. 1997. Physical Chemistry of Surfaces. Wiley.Google Scholar
    2. Adelson, E. H. and J. R. Bergen. 1991. The plenoptic function and the elements of early vision. Computational models of visual processing 1, 2 (1991).Google Scholar
    3. Antipa, N., S. Necula, R. Ng, and L. Waller. 2016. Single-Shot Diffuser-Encoded Light Field Imaging. In IEEE International Conference on Computational Photography (ICCP). Google ScholarCross Ref
    4. Barnum, P. C., S. G. Narasimhan, and T. Kanade. 2010. A Multi-layered Display with Water Drops. ACM Trans. Graph. 29, 4, Article 76 (2010), 7 pages. Google ScholarDigital Library
    5. Bouguet, J.-Y. 2004. Camera calibration toolbox for MATLAB. (2004).Google Scholar
    6. Brakke, K. A. 1992. The surface evolver. Experimental mathematics 1, 2 (1992), 141–165. Google ScholarCross Ref
    7. Brakke, K. A. 2013. Surface Evolver 2.70. (2013). http://facstaff.susqu.edu/brakke/evolver/evolver.html.Google Scholar
    8. Chai, J.-X., X. Tong, S.-C. Chan, and H.-Y. Shum. 2000. Plenoptic Sampling. In Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’00). 307–318. Google ScholarDigital Library
    9. Chronis, N., G. L. Liu, K.-H. Jeong, and L. P. Lee. 2003. Tunable liquid-filled microlens array integrated with microfluidic network. Opt. Express 11, 19 (Sep 2003), 2370–2378. Google ScholarCross Ref
    10. Cimpoi, M., S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. 2014. Describing Textures in the Wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). Google ScholarDigital Library
    11. Davis, A., M. Levoy, and F. Durand. 2012. Unstructured Light Fields. Comp. Graph. Forum 31, 2 (May 2012), 305–314. Google ScholarDigital Library
    12. De Gennes, P.-G., F. Brochard-Wyart, and D. Quéré. 2004. Capillarity and wetting phenomena: drops, bubbles, pearls, waves. Springer Science & Business Media. Google ScholarCross Ref
    13. Eigen, D., D. Krishnan, and R. Fergus. 2013. Restoring an Image Taken through a Window Covered with Dirt or Rain. In IEEE International Conference on Computer Vision (ICCV). IEEE, 633–640. Google ScholarDigital Library
    14. Fergus, R., A. Torralba, and W. T. Freeman. 2006. Random lens imaging. Technical Report MIT-CSAIL-TR-2006–058.Google Scholar
    15. Fuchs, M., M. Kächele, and S. Rusinkiewicz. 2013. Design and Fabrication of Faceted Mirror Arrays for Light Field Capture. Computer Graphics Forum 32, 8 (2013), 246–257. Google ScholarCross Ref
    16. Georgiev, T., K. C. Zheng, B. Curless, D. Salesin, S. Nayar, and C. Intwala. 2006. Spatio-angular Resolution Tradeoffs in Integral Photography. In Proceedings of the 17th Eurographics Conference on Rendering Techniques (EGSR ’06). Eurographics Association, 263–272. Google ScholarCross Ref
    17. Goldlücke, B., E. Strekalovskiy, and D. Cremers. 2012. The natural vectorial total variation which arises from geometric measure theory. SIAM Journal on Imaging Sciences 5, 2 (2012), 537–563. Google ScholarCross Ref
    18. Gortler, S. J., R. Grzeszczuk, R. Szeliski, and M. F. Cohen. 1996. The Lumigraph. In Proc. 23rd Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’96). ACM, New York, NY, USA, 43–54. Google ScholarDigital Library
    19. Gulshan, V., C. Rother, A. Criminisi, A. Blake, and A. Zisserman. 2010. Geodesic star convexity for interactive image segmentation. In Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on. IEEE, 3129–3136. Google ScholarCross Ref
    20. Han, J. Y. and K. Perlin. 2003. Measuring Bidirectional Texture Reflectance with a Kaleidoscope. ACM Trans. Graph. (Proc. SIGGRAPH 2003) (2003), 741–748. Google ScholarDigital Library
    21. Hartley, R. I. and A. Zisserman. 2004. Multiple View Geometry in Computer Vision (2nd ed.). Cambridge University Press. Google ScholarCross Ref
    22. Hickson, P., E. F. Borra, R. Cabanac, S. C. Chapman, V. De Lapparent, M. Mulrooney, and G. A. Walker. 1998. Large Zenith Telescope project: a 6-m mercury-mirror telescope. In Astronomical Telescopes & Instrumentation. International Society for Optics and Photonics, 226–232.Google Scholar
    23. Honauer, K., O. Johannsen, D. Kondermann, and B. Goldluecke. 2016. A dataset and evaluation methodology for depth estimation on 4D light fields. In Asian Conference on Computer Vision. Springer.Google Scholar
    24. Hullin, M. B., M. Fuchs, I. Ihrke, H.-P. Seidel, and H. P. A. Lensch. 2008. Fluorescent Immersion Range Scanning. ACM Trans. Graph. (Proc. SIGGRAPH 2008) 27, 3 (Aug. 2008), 87:1–87:10. Google ScholarDigital Library
    25. Hullin, M. B., H. P. A. Lensch, R. Raskar, H.-P. Seidel, and I. Ihrke. 2011. Dynamic Display of BRDFs. In Computer Graphics Forum (Proc. EUROGRAPHICS), Oliver Deussen and Min Chen (Eds.). Eurographics, Blackwell, Llandudno, UK, 475–483. Google ScholarDigital Library
    26. Ihrke, I., K. Kutulakos, H. Lensch, M. Magnor, and W. Heidrich. 2008. State of the art in transparent and specular object reconstruction. In EUROGRAPHICS 2008 STAR.Google Scholar
    27. Ihrke, I., G. Wetzstein, D. Lanman, and W. Heidrich. 2011. State of the art in computational plenoptic imaging. In EUROGRAPHICS 2011 STAR.Google Scholar
    28. Iliev, S. 1995. Iterative method for the shape of static drops. Computer Methods in Applied Mechanics and Engineering 126, 3 (1995), 251–265. Google ScholarCross Ref
    29. Iliev, S. 1997. Static drops on an inclined plane: equilibrium modeling and numerical analysis. Journal of colloid and interface science 194, 2 (1997), 287–300. Google ScholarCross Ref
    30. Iliev, S. and N. Pesheva. 2003. Wetting properties of well-structured heterogeneous substrates. Langmuir 19, 23 (2003), 9923–9931. Google ScholarCross Ref
    31. Iliev, S. and N. Pesheva. 2006. Nonaxisymmetric drop shape analysis and its application for determination of the local contact angles. Journal of colloid and interface science 301, 2 (2006), 677–684. Google ScholarCross Ref
    32. Jakob, W. 2010. Mitsuba renderer. (2010). http://www.mitsuba-renderer.org.Google Scholar
    33. Kim, C., H. Zimmer, Y. Pritch, A. Sorkine-Hornung, and M. Gross. 2013. Scene Reconstruction from High Spatio-Angular Resolution Light Fields. ACM Trans. Graph. (Proc. SIGGRAPH 2013) 32, 4 (2013), 73:1–73:12.Google Scholar
    34. Kuiper, S. and B. Hendriks. 2004. Variable-focus liquid lens for miniature cameras. Applied Physics Petters 85, 7 (2004), 1128–1130. Google ScholarCross Ref
    35. Kutulakos, K. N. and E. Steger. 2008. A Theory of Refractive and Specular 3D Shape by Light-Path Triangulation. International Journal of Computer Vision 76, 1 (2008), 13–29. Google ScholarDigital Library
    36. Levoy, M. and P. Hanrahan. 1996. Light Field Rendering. In Proc. 23rd Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’96). ACM, New York, NY, USA, 31–42. Google ScholarDigital Library
    37. Levoy, M., R. Ng, A. Adams, M. Footer, and M. Horowitz. 2006. Light Field Microscopy. ACM Trans. Graph. (Proc. SIGGRAPH 2013) (2006), 924–934. Google ScholarDigital Library
    38. Lippmann, G. 1908. La photographie intégrale. CR Acad. Sci. 146 (1908), 446–451.Google Scholar
    39. Lowe, D. G. 1999. Object recognition from local scale-invariant features. In IEEE International Conference on Computer Vision (ICCV). 1150–1157. Google ScholarCross Ref
    40. Manakov, A., J. F. Restrepo, O. Klehm, R. Hegedüs, E. Eisemann, H.-P. Seidel, and I. Ihrke. 2013. A Reconfigurable Camera Add-on for High Dynamic Range, Multi-Spectral, Polarization, and Light-Field Imaging. ACM Trans. Graph. (Proc. SIGGRAPH 2013) 32, 4, Article 47 (July 2013), 14 pages. Google ScholarDigital Library
    41. Mukaigawa, Y., S. Tagawa, J. Kim, R. Raskar, Y. Matsushita, and Y. Yagi. 2011. Hemispherical Confocal Imaging Using Turtleback Reflector. In Computer Vision – ACCV 2010. Springer, 336–349. Google ScholarCross Ref
    42. Ng, R. 2005. Fourier Slice Photography. ACM Trans. Graph. (Proc. SIGGRAPH 2005) (2005), 735–744. Google ScholarDigital Library
    43. O’Neill, F. T. and J. T. Sheridan. 2002. Photoresist reflow method of microlens production Part I: Background and experiments. Optik-International Journal for Light and Electron Optics 113, 9 (2002), 391–404. Google ScholarCross Ref
    44. Pock, T., D. Cremers, H. Bischof, and A. Chambolle. 2010. Global Solutions of Variational Models with Convex Regularization. SIAM Journal on Imaging Sciences (2010).Google Scholar
    45. Raskar, R., A. Agrawal, C. A. Wilson, and A. Veeraraghavan. 2008. Glare Aware Photography: 4D Ray Sampling for Reducing Glare Effects of Camera Lenses. ACM Trans. Graph. (Proc. SIGGRAPH 2008) 27, 3, Article 56 (Aug. 2008), 10 pages. Google ScholarDigital Library
    46. Shan, Q., B. Curless, and T. Kohno. 2010. Seeing Through Obscure Glass. In Proceedings of the 11th European Conference on Computer Vision: Part VI (ECCV’10). Springer-Verlag, Berlin, Heidelberg, 364–378. http://dl.acm.org/citation.cfm?id=1888212.1888241 Google ScholarCross Ref
    47. Taguchi, Y., A. Agrawal, A. Veeraraghavan, S. Ramalingam, and R. Raskar. 2010. Axial-Cones: Modeling Spherical Catadioptric Cameras for Wide-Angle Light Field Rendering. ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia 2010) 29, 6 (Dec 2010), 172:1–172:8.Google Scholar
    48. Tao, M. W., S. Hadap, J. Malik, and R. Ramamoorthi. 2013. Depth from Combining Defocus and Correspondence Using Light-Field Cameras. In IEEE International Conference on Computer Vision (ICCV). 673–680. Google ScholarDigital Library
    49. Tarini, M., H. P. A. Lensch, M. Goesele, and H.-P. Seidel. 2005. 3D acquisition of mirroring objects using striped patterns. Graphical Models 67, 4 (2005), 233–259. Google ScholarDigital Library
    50. Torralba, A. and W. Freeman. 2014. Accidental Pinhole and Pinspeck Cameras. International Journal of Computer Vision 110, 2 (2014), 92–112. Google ScholarDigital Library
    51. Vaish, V. and others. 2008. The (New) Stanford Light Field Archive. (2008). http://lightfield.stanford.edu/lfs.html.Google Scholar
    52. Veeraraghavan, A., R. Raskar, A. Agrawal, A. Mohan, and J. Tumblin. 2007. Dappled Photography: Mask Enhanced Cameras for Heterodyned Light Fields and Coded Aperture Refocusing. ACM Trans. Graph. (Proc. SIGGRAPH 2007) 26, 3, Article 69 (2007). Google ScholarDigital Library
    53. Wang, T.-C., A. Efros, and R. Ramamoorthi. 2016. Depth estimation with occlusion modeling using light-field cameras. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) (2016).Google Scholar
    54. Wanner, S. and B. Goldlücke. 2014. Variational Light Field Analysis for Disparity Estimation and Super-Resolution. IEEE Transactions on Pattern Analysis and Machine Intelligence 36, 3 (2014), 606–619. Google ScholarDigital Library
    55. Wei, L.-Y., C.-K. Liang, G. Myhre, C. Pitts, and K. Akeley. 2015. Improving Light Field Camera Sample Design with Irregularity and Aberration. ACM Trans. Graph. 34, 4, Article 152 (2015), 11 pages. Google ScholarDigital Library
    56. Weinmann, M., A. Osep, R. Ruiters, and R. Klein. 2013. Multi-View Normal Field Integration for 3D Reconstruction of Mirroring Objects. Proceedings of the International Conference on Computer Vision (Dec. 2013), 2504–2511. Google ScholarDigital Library
    57. Wender, A., J. Iseringhausen, B. Goldlücke, M. Fuchs, and M. B. Hullin. 2015. Light Field Imaging through Household Optics. In Vision, Modeling & Visualization, David Bommes, Tobias Ritschel, and Thomas Schultz (Eds.). Eurographics Association, 159–166. Google ScholarCross Ref
    58. Wetzstein, G., I. Ihrke, and W. Heidrich. 2013. On Plenoptic Multiplexing and Reconstruction. International Journal of Computer Vision 101, 2 (2013), 384–400. Google ScholarDigital Library
    59. Wilburn, B., N. Joshi, V. Vaish, E.-V. Talvala, E. Antunez, A. Barth, A. Adams, M. Horowitz, and M. Levoy. 2005. High Performance Imaging Using Large Camera Arrays. ACM Trans. Graph. (Proc. SIGGRAPH 2005) (2005), 765–776. Google ScholarDigital Library
    60. You, S., R. T. Tan, R. Kawakami, Y. Mukaigawa, and K. Ikeuchi. 2016. Waterdrop Stereo. CoRR (2016). arXiv:1604.00730v1Google Scholar

ACM Digital Library Publication:


Overview Page: