“Local Fourier Slice Photography” by Lessig
Conference:
Type:
Title:
- Local Fourier Slice Photography
Session/Category Title: RGB++: Depth, 3D, and Light Fields
Presenter(s)/Author(s):
Abstract:
Light field cameras provide intriguing possibilities, such as post-capture refocus or the ability to synthesize images from novel viewpoints. This comes, however, at the price of significant storage requirements. Compression techniques can be used to reduce these, but refocusing and reconstruction require so far, again, a dense pixel representation. To avoid this, we introduce local Fourier slice photography that allows for refocused image reconstruction directly from a sparse wavelet representation of a light field, either to obtain an image or a compressed representation of it. The result is made possible by wavelets that respect the “slicing’s” intrinsic structure and enable us to derive exact reconstruction filters for the refocused image in closed-form. Image reconstruction then amounts to applying these filters to the light field’s wavelet coefficients; hence, no reconstruction of a dense pixel representation is required. We demonstrate that this can reduce storage requirements and also computation times. We furthermore analyze the computational complexity of our algorithm and show that it scales linearly with the size of the reconstructed region and the non-negligible wavelet coefficients, i.e., with the visual complexity.
References:
1. A. Aggoun. 2006. A 3D Dct compression algorithm for omnidirectional integral images. In 2006 IEEE International Conference on Acoustics Speed and Signal Processing Proceedings, Vol. 2. IEEE, II–517–II–520. DOI:https://doi.org/10.1109/ICASSP.2006.1660393
2. A. Aggoun and M. Mazri. 2008. Wavelet-based compression algorithm for still omnidirectional 3d integral images. Signal, Image and Video Processing 2, 2 (June 2008), 141–153. DOI:https://doi.org/10.1007/s11760-007-0044-1
3. G. Alves, M. P. Pereira, M. B. de Carvalho, F. Pereira, C. L. Pagliari, V. Testoni, and E. A.B. da Silva. 2018. A study on the 4D sparsity of JPEG pleno light fields using the discrete cosine transform. In 2018 25th IEEE International Conference on Image Processing (ICIP). IEEE, 1148–1152. DOI:https://doi.org/10.1109/ICIP.2018.8451583
4. L. Belcour, C. Soler, K. Subr, N. Holzschuch, and F. Durand. 2013. 5D covariance tracing for efficient defocus and motion blur. ACM Transactions on Graphics 32, 3 (June 2013), 1–18. DOI:https://doi.org/10.1145/2487228.2487239
5. E. Candès and D. L. Donoho. 2004. New tight frames of curvelets and optimal representations of objects with piecewise C2 singularities. Communications on Pure and Applied Mathematics 57, 2 (Feb 2004), 219–266. DOI:https://doi.org/10.1002/cpa.10116
6. E. J. Candès and D. L. Donoho. 1999. Ridgelets: A key to higher-dimensional intermittency? Royal Society of London Philosophical Transactions Series A 357 (1999), 2495–+.
7. E. J. Candès and D. L. Donoho. 2005a. Continuous curvelet transform: I. Resolution of the Wavefront Set. Applied and Computational Harmonic Analysis 19, 2 (Sept. 2005), 162–197. DOI:https://doi.org/10.1016/j.acha.2005.02.003
8. E. J. Candès and D. L. Donoho. 2005b. Continuous curvelet transform: II. Discretization and Frames. Applied and Computational Harmonic Analysis 19, 2 (Sept. 2005), 198–222. DOI:https://doi.org/10.1016/j.acha.2005.02.004
9. J.-X. Chai, S.-C. Chan, H.-Y. Shum, and X. Tong. 2000. Plenoptic sampling. In Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH’00). ACM Press, New York, New York, 307–318. DOI:https://doi.org/10.1145/344779.344932
10. C. Conti, P. T. Kovacs, T. Balogh, P. Nunes, and L. D. Soares. 2014. Light-field video coding using geometry-based disparity compensation. In 2014 3DTV-Conference: The True Vision—Capture, Transmission and Display of 3D Video (3DTV-CON). IEEE, 1–4. DOI:https://doi.org/10.1109/3DTV.2014.6874724
11. F. Dai, J. Zhang, Y. Ma, and Y. Zhang. 2015. Lenselet image compression scheme based on subaperture images streaming. In 2015 IEEE International Conference on Image Processing (ICIP). IEEE, 4733–4737. DOI:https://doi.org/10.1109/ICIP.2015.7351705
12. I. Daubechies. 1992. Ten Lectures on Wavelets. Society for Industrial and Applied Mathematics, Philadelphia, PA.
13. R. A. DeVore. 2006. Optimal computation. In Proceedings of the International Congress of Mathematicians. Madrid, Spain.
14. P. Didyk, P. Sitthi-Amorn, W. Freeman, F. Durand, and W. Matusik. 2013. Joint view expansion and filtering for automultiscopic 3D displays. ACM Transactions on Graphics 32, 6 (Nov. 2013), 1–8. DOI:https://doi.org/10.1145/2508363.2508376
15. D. L. Donoho. 2000. Orthonormal ridgelets and linear singularities. SIAM Journal on Mathematical Analysis 31, 5 (Aug. 2000), 1062. DOI:https://doi.org/10.1137/S0036141098344403
16. F. Durand, N. Holzschuch, C. Soler, E. Chan, and F. Sillion. 2005. A frequency analysis of light transport. ACM Transactions on Graphics (Proceedings of SIGGRAPH 2005) 24, 3 (2005), 1115–1126.
17. W. T. Freeman and E. H. Adelson. 1991. The design and use of steerable filters. IEEE Transactions on Pattern Analysis and Machine Intelligence 13, 9 (1991), 891–906. DOI:https://doi.org/10.1109/34.93808
18. E. S. L. Gastal and M. M. Oliveira. 2017. Spectral remapping for image downscaling. ACM Transactions on Graphics 36, 4 (July 2017), 1–16. DOI:https://doi.org/10.1145/3072959.3073670
19. S. J. Gortler, R. Grzeszczuk, R. Szeliski, and M. F. Cohen. 1996. The lumigraph. In SIGGRAPH’96: Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques. ACM, New York, NY, 43–54. DOI:https://doi.org/10.1145/237170.237200
20. I. Ihrke, J. Restrepo, and L. Mignard-Debise. 2016. Principles of light field imaging: Briefly revisiting 25 years of research. IEEE Signal Processing Magazine 33, 5 (Sept. 2016), 59–69. DOI:https://doi.org/10.1109/MSP.2016.2582220
21. N. K. Kalantari, T.-C. Wang, and R. Ramamoorthi. 2016. Learning-based view synthesis for light field cameras. ACM Transactions on Graphics 35, 6 (Nov. 2016), 1–10. DOI:https://doi.org/10.1145/2980179.2980251
22. P. Kellnhofer, P. Didyk, S.-P. Wang, P. Sitthi-Amorn, W. Freeman, F. Durand, and W. Matusik. 2017. 3DTV at home. ACM Transactions on Graphics 36, 4 (July 2017), 1–13. DOI:https://doi.org/10.1145/3072959.3073617
23. S. Kundu. 2012. Light field compression using homography and 2D warping. In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 1349–1352. DOI:https://doi.org/10.1109/ICASSP.2012.6288140
24. J. Lehtinen, T. Aila, J. Chen, S. Laine, and F. Durand. 2011. Temporal light field reconstruction for rendering distribution effects. In ACM SIGGRAPH 2011 Papers on—SIGGRAPH’11, Vol. 30. ACM Press, New York, New York, 1. DOI:https://doi.org/10.1145/1964921.1964950
25. C. Lessig. 2018a. A local Fourier slice equation. Optics Express 26, 23 (Nov. 2018), 29769. DOI:https://doi.org/10.1364/OE.26.029769
26. C. Lessig. 2018b. Polar Wavelets in Space. Technical Report. arxiv:1805.02061 https://arxiv.org/abs/1805.02061
27. A. Levin, W. T. Freeman, and F. Durand. 2008. Understanding camera trade-offs through a Bayesian analysis of light field projections. In Computer Vision â ECCV 2008. Springer Berlin, Germany, 88–101. DOI:https://doi.org/10.1007/978-3-540-88693-8_7
28. M. Levoy and P. Hanrahan. 1996. Light field rendering. In SIGGRAPH’96: Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques. ACM, New York, NY, 31–42. DOI:https://doi.org/10.1145/237170.237199
29. M. Levoy, R. Ng, A. Adams, M. Footer, and M. Horowitz. 2006. Light field microscopy. In ACM Trans. Graph. (Proceedings of ACM SIGGRAPH 2006), Vol. 25. ACM Press, New York, New York, 924. DOI:https://doi.org/10.1145/1179352.1141976
30. C.-K. Liang and R. Ramamoorthi. 2015. A light transport framework for lenslet light field cameras. ACM Transactions on Graphics 34, 2 (Mar. 2015), 1–19. DOI:https://doi.org/10.1145/2665075
31. S. G. Mallat. 2009. A Wavelet Tour of Signal Processing: The Sparse Way (3rd ed.). Academic Press.
32. R. Ng. 2005. Fourier slice photography. ACM Transactions on Graphics 24, 3 (July 2005), 735. DOI:https://doi.org/10.1145/1073204.1073256
33. R. Ng, M. Levoy, M. Brédif, G. Duval, M. Horowitz, and P. Hanrahan. 2005. Light Field Photography with a Hand-Held Plenoptic Camera. Technical report, Stanford University.
34. P. Perona. 1991. Deformable kernels for early vision. In Proceedings of the 1991 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE, 222–227. DOI:https://doi.org/10.1109/CVPR.1991.139691
35. M. Pharr and G. Humphreys. 2010. Physically Based Rendering: From Theory to Implementation (2nd ed.). Morgan Kaufmann Publishers Inc., San Francisco, CA.
36. J. Portilla and E. P. Simoncelli. 2000. A parametric texture model based on joint statistics of complex wavelet coefficients. International Journal of Computer Vision 40, 1 (2000), 49–70. DOI:https://doi.org/10.1023/A:1026553619983
37. E. T. Quinto. 1993. Singularities of the X-ray transform and limited data tomography in R2 and R3. SIAM Journal on Mathematical Analysis 24, 5 (Sept. 1993), 1215–1225. DOI:https://doi.org/10.1137/0524069
38. E. T. Quinto. 2007. Local algorithms in exterior tomography. J. Comput. Appl. Math. 199, 1 (Feb. 2007), 141–148. DOI:https://doi.org/10.1016/J.CAM.2004.11.055
39. P. Sen, S. Darabi, and L. Xiao. 2011. Compressive rendering of multidimensional scenes. Springer Berlin, Germany, 152–183. DOI:https://doi.org/10.1007/978-3-642-24870-2_7
40. L. Shi, H. Hassanieh, A. Davis, D. Katabi, and F. Durand. 2014. Light field reconstruction using sparsity in the continuous Fourier domain. ACM Transactions on Graphics 34, 1 (Dec. 2014), 1–13. DOI:https://doi.org/10.1145/2682631
41. E. P. Simoncelli and W. T. Freeman. 1995. The steerable pyramid: A flexible architecture for multi-scale derivative computation. In Proceedings of the International Conference on Image Processing, Vol. 3. IEEE Comput. Soc. Press, 444–447. DOI:https://doi.org/10.1109/ICIP.1995.537667
42. C. Soler, K. Subr, F. Durand, N. Holzschuch, and F. Sillion. 2009. Fourier depth of field. ACM Transactions on Graphics 28, 2 (Apr. 2009), 1–12. DOI:https://doi.org/10.1145/1516522.1516529
43. R. Stevenson. 2009. Adaptive wavelet methods for solving operator equations: An overview. In Multiscale, Nonlinear and Adaptive Approximation. Springer Berlin, Germany, 543–597. DOI:https://doi.org/10.1007/978-3-642-03413-8_13
44. M. Unser and N. Chenouard. 2013. A unifying parametric framework for 2D steerable wavelet transforms. SIAM Journal on Imaging Sciences 6, 1 (Jan. 2013), 102–135. DOI:https://doi.org/10.1137/120866014
45. M. Unser, N. Chenouard, and D. Van De Ville. 2011. Steerable pyramids and tight wavelet frames in L2(Rd). IEEE Transactions on Image Processing 20, 10 (Oct. 2011), 2705–2721. DOI:https://doi.org/10.1109/TIP.2011.2138147
46. M. Unser and D. Van De Ville. 2010. Wavelet steerability and the higher-order Riesz transform. IEEE Transactions on Image Processing 19, 3 (Mar. 2010), 636–652. DOI:https://doi.org/10.1109/TIP.2009.2038832
47. S. Vagharshakyan, R. Bregovic, and A. Gotchev. 2018. Light field reconstruction using shearlet transform. IEEE Transactions on Pattern Analysis and Machine Intelligence 40, 1 (Jan. 2018), 133–147. DOI:https://doi.org/10.1109/TPAMI.2017.2653101
48. K. Venkataraman, D. Lelescu, J. Duparré, A. McMahon, G. Molina, P. Chatterjee, R. Mullis, and S. Nayar. 2013. PiCam: An ultra-thin high performance monolithic camera array. ACM Transactions on Graphics 32, 6 (Nov. 2013), 1–13. DOI:https://doi.org/10.1145/2508363.2508390
49. A. Vieira, H. Duarte, C. Perra, L. Tavora, and P. Assuncao. 2015. Data formats for high efficiency coding of Lytro-Illum light fields. In 2015 International Conference on Image Processing Theory, Tools and Applications (IPTA). IEEE, 494–497. DOI:https://doi.org/10.1109/IPTA.2015.7367195
50. I. Viola, M. Rerabek, and T. Ebrahimi. 2017. Comparison and evaluation of light field image coding approaches. IEEE Journal of Selected Topics in Signal Processing 11, 7 (Oct. 2017), 1092–1106. DOI:https://doi.org/10.1109/JSTSP.2017.2740167
51. N. Wadhwa, M. Rubinstein, F. Durand, and W. T. Freeman. 2013. Phase-based video motion processing. ACM Transactions on Graphics 32, 4 (July 2013), 1. DOI:https://doi.org/10.1145/2461912.2461966
52. J. P. Ward, P. Pad, and M. Unser. 2015. Optimal isotropic wavelets for localized tight frame representations. IEEE Signal Processing Letters 22, 11 (Nov. 2015), 1918–1921. DOI:https://doi.org/10.1109/LSP.2015.2448233
53. J. P. Ward and M. Unser. 2014. Harmonic singular integrals and steerable wavelets in L2(Rd). Applied and Computational Harmonic Analysis 36, 2 (Mar. 2014), 183–197. DOI:https://doi.org/10.1016/J.ACHA.2013.03.006
54. B. Wilburn, N. Joshi, V. Vaish, E.-V. Talvala, E. Antunez, A. Barth, A. Adams, M. Horowitz, and M. Levoy. 2005. High performance imaging using large camera arrays. ACM Trans. Graph. (Proceedings of ACM SIGGRAPH 2005) (2005). https://graphics.stanford.edu/papers/CameraArray/.
55. G. Wu, B. Masia, A. Jarabo, Y. Zhang, L. Wang, Q. Dai, T. Chai, and Y. Liu. 2017. Light field image processing: An overview. IEEE Journal of Selected Topics in Signal Processing 11, 7 (Oct. 2017), 926–954. DOI:https://doi.org/10.1109/JSTSP.2017.2747126
56. Z. Xu, K. Sunkavalli, S. Hadap, and R. Ramamoorthi. 2018. Deep image-based relighting from optimal sparse samples. ACM Transactions on Graphics 37, 4 (July 2018), 1–13. DOI:https://doi.org/10.1145/3197517.3201313
57. Y. Yoon, H.-G. Jeon, D. Yoo, J.-Y. Lee, and I. S. Kweon. 2015. Learning a deep convolutional network for light-field image super-resolution. In 2015 IEEE International Conference on Computer Vision Workshop (ICCVW). IEEE, 57–65. DOI:https://doi.org/10.1109/ICCVW.2015.17
