“Learned large field-of-view imaging with thin-plate optics” by Peng, Sun, Dun, Wetzstein, Heidrich, et al. …
Conference:
Type(s):
Title:
- Learned large field-of-view imaging with thin-plate optics
Session/Category Title: Light Hardware
Presenter(s)/Author(s):
Moderator(s):
Abstract:
Typical camera optics consist of a system of individual elements that are designed to compensate for the aberrations of a single lens. Recent computational cameras shift some of this correction task from the optics to post-capture processing, reducing the imaging optics to only a few optical elements. However, these systems only achieve reasonable image quality by limiting the field of view (FOV) to a few degrees – effectively ignoring severe off-axis aberrations with blur sizes of multiple hundred pixels.In this paper, we propose a lens design and learned reconstruction architecture that lift this limitation and provide an order of magnitude increase in field of view using only a single thin-plate lens element. Specifically, we design a lens to produce spatially shift-invariant point spread functions, over the full FOV, that are tailored to the proposed reconstruction architecture. We achieve this with a mixture PSF, consisting of a peak and and a low-pass component, which provides residual contrast instead of a small spot size as in traditional lens designs. To perform the reconstruction, we train a deep network on captured data from a display lab setup, eliminating the need for manual acquisition of training data in the field. We assess the proposed method in simulation and experimentally with a prototype camera system.We compare our system against existing single-element designs, including an aspherical lens and a pinhole, and we compare against a complex multielement lens, validating high-quality large field-of-view (i.e. 53°) imaging performance using only a single thin-plate element.
References:
1. Se Hyun Ahn and L Jay Guo. 2009. Large-area roll-to-roll and roll-to-plate nanoimprint lithography: a step toward high-throughput application of continuous nanoimprinting. ACS Nano 3, 8 (2009), 2304–2310.Google ScholarCross Ref
2. Nick Antipa, Grace Kuo, Reinhard Heckel, Ben Mildenhall, Emrah Bostan, Ren Ng, and Laura Waller. 2018. DiffuserCam: lensless single-exposure 3D imaging. Optica 5, 1 (2018), 1–9.Google ScholarCross Ref
3. Martin Arjovsky, Soumith Chintala, and Léon Bottou. 2017. Wasserstein gan. arXiv preprint arXiv:1701.07875 (2017).Google Scholar
4. Yoshua Bengio. 2012. Deep learning of representations for unsupervised and transfer learning. In Proceedings of ICML Workshop on Unsupervised and Transfer Learning. 17–36.Google Scholar
5. Glenn D Boreman. 2001. Modulation transfer function in optical and electro-optical systems. Vol. 21. SPIE press Bellingham, WA.Google Scholar
6. David J Brady, Michael E Gehm, Ronald A Stack, Daniel L Marks, David S Kittle, Dathon R Golish, EM Vera, and Steven D Feller. 2012. Multiscale gigapixel photography. Nature 486, 7403 (2012), 386.Google ScholarCross Ref
7. Tim Brooks, Ben Mildenhall, Tianfan Xue, Jiawen Chen, Dillon Sharlet, and Jonathan T Barron. 2018. Unprocessing Images for Learned Raw Denoising. arXiv preprint arXiv:1811.11127 (2018).Google Scholar
8. Julie Chang, Vincent Sitzmann, Xiong Dun, Wolfgang Heidrich, and Gordon Wetzstein. 2018b. Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification. Scientific reports 8, 1 (2018), 12324.Google Scholar
9. Julie Chang and Gordon Wetzstein. 2019. Deep Optics for Monocular Depth Estimation and 3D Object Detection. arXiv preprint arXiv:1904.08601 (2019).Google Scholar
10. Li-Wen Chang, Yang Chen, Wenlei Bao, Amit Agarwal, Eldar Akchurin, Ke Deng, and Emad Barsoum. 2018a. Accelerating Recurrent Neural Networks through Compiler Techniques and Quantization. (2018).Google Scholar
11. Chen Chen, Qifeng Chen, Jia Xu, and Vladlen Koltun. 2018. Learning to See in the Dark. (2018).Google Scholar
12. Taeg Sang Cho, Charles L Zitnick, Neel Joshi, Sing Bing Kang, Richard Szeliski, and William T Freeman. 2012. Image restoration by matching gradient distributions. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 34, 4 (2012), 683–694.Google ScholarDigital Library
13. Stephen Y Chou, Peter R Krauss, and Preston J Renstrom. 1996. Nanoimprint lithography. Journal of Vacuum Science & Technology B: Microelectronics and Nanometer Structures Processing, Measurement, and Phenomena 14, 6 (1996), 4129–4133.Google ScholarCross Ref
14. Oliver Cossairt and Shree Nayar. 2010. Spectral focal sweep: Extended depth of field from chromatic aberrations. In Computational Photography (ICCP), IEEE International Conference on. IEEE, 1–8.Google Scholar
15. Edward R Dowski and W Thomas Cathey. 1995. Extended depth of field through wave-front coding. Applied optics 34, 11 (1995), 1859–1866.Google Scholar
16. Wang Duoshu, Chongtai Luo, Yuqing Xiong, Tao Chen, Hongkai Liu, and Jizhou Wang. 2011. Fabrication technology of the centrosymmetric continuous relief diffractive optical elements. Physics Procedia 18 (2011), 95–99.Google ScholarCross Ref
17. EMVA Standard. 2005. 1288: Standard for characterization and presentation of specification data for image sensors and cameras. European Machine Vision Association (2005).Google Scholar
18. Magali Estribeau and Pierre Magnan. 2004. Fast MTF measurement of CMOS imagers using ISO 12333 slanted-edge methodology. In Detectors and Associated Signal Processing, Vol. 5251. International Society for Optics and Photonics, 243–253.Google ScholarCross Ref
19. FZ Fang, XD Zhang, A Weckenmann, GX Zhang, and C Evans. 2013. Manufacturing and measurement of freeform optics. CIRP Annals 62, 2 (2013), 823–846.Google ScholarCross Ref
20. Grant R Fowles. 2012. Introduction to modern optics. Courier Dover Publications.Google Scholar
21. Carl Friedrich Gauss. 1843. Dioptrische Untersuchungen von CF Gauss. in der Dieterichschen Buchhandlung.Google Scholar
22. Joseph M Geary. 2002. Introduction to lens design: with practical ZEMAX examples. Willmann-Bell Richmond.Google Scholar
23. Marc Geese, Ulrich Seger, and Alfredo Paolillo. 2018. Detection Probabilities: Performance Prediction for Sensors of Autonomous Vehicles. Electronic Imaging 2018, 17 (2018), 148-1–148-14.Google ScholarCross Ref
24. Patrice Genevet, Federico Capasso, Francesco Aieta, Mohammadreza Khorasaninejad, and Robert Devlin. 2017. Recent advances in planar optics: from plasmonic to dielectric metasurfaces. Optica 4, 1 (2017), 139–152.Google ScholarCross Ref
25. Erez Gilad and Jost Von Hardenberg. 2006. A fast algorithm for convolution integrals with space and time variant kernels. J. Comput. Phys. 216, 1 (2006), 326–336.Google ScholarDigital Library
26. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in neural information processing systems. 2672–2680.Google Scholar
27. Joseph W Goodman. 2005. Introduction to Fourier optics. Roberts and Company Publishers.Google Scholar
28. Hrel Haim, Shay Elmalem, Raja Giryes, Alex M. Bronstein, and Emanuel Marom. 2018. Depth Estimation From a Single Image Using Deep Learned Phase Coded Mask. IEEE Trans. Computational Imaging 4, 3 (2018), 298–310.Google ScholarCross Ref
29. Samuel W Hasinoff, Dillon Sharlet, Ryan Geiss, Andrew Adams, Jonathan T Barron, Florian Kainz, Jiawen Chen, and Marc Levoy. 2016. Burst photography for high dynamic range and low-light imaging on mobile cameras. ACM Transactions on Graphics (TOG) 35, 6 (2016), 192.Google ScholarDigital Library
30. Eugene Hecht. 1998. Hecht optics. Addison Wesley 997 (1998), 213–214.Google Scholar
31. Felix Heide, Qiang Fu, Yifan Peng, and Wolfgang Heidrich. 2016. Encoded diffractive optics for full-spectrum computational imaging. Scientific Reports 6 (2016).Google Scholar
32. Felix Heide, Mushfiqur Rouf, Matthias B Hullin, Bjorn Labitzke, Wolfgang Heidrich, and Andreas Kolb. 2013. High-quality computational imaging through simple lenses. ACM Transactions on Graphics (TOG) 32, 5 (2013), 149.Google ScholarDigital Library
33. Felix Heide, Markus Steinberger, Yun-Ta Tsai, Mushfiqur Rouf, Dawid Pająk, Dikpal Reddy, Orazio Gallo, Jing Liu, Wolfgang Heidrich, Karen Egiazarian, et al. 2014. Flexisp: A flexible camera image processing framework. ACM Transactions on Graphics (TOG) 33, 6 (2014), 231.Google ScholarDigital Library
34. Minyoung Huh, Pulkit Agrawal, and Alexei A Efros. 2016. What makes ImageNet good for transfer learning? arXiv preprint arXiv:1608.08614 (2016).Google Scholar
35. Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017. Image-to-Image Translation with Conditional Adversarial Networks. CVPR (2017).Google Scholar
36. Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016. Perceptual losses for real-time style transfer and super-resolution. In European Conference on Computer Vision. Springer, 694–711.Google ScholarCross Ref
37. Arnold Kalvach and Zsolt Szabó. 2016. Aberration-free flat lens design for a wide range of incident angles. Journal of the Optical Society of America B 33, 2 (2016), A66.Google ScholarCross Ref
38. Rudolf Kingslake and R Barry Johnson. 2009. Lens design fundamentals. Academic Press.Google Scholar
39. Dilip Krishnan and Rob Fergus. 2009. Fast image deconvolution using hyper-Laplacian priors. In Advances in Neural Information Processing Systems. NIPS, 1033–1041.Google Scholar
40. Orest Kupyn, Volodymyr Budzan, Mykola Mykhailych, Dmytro Mishkin, and Jiri Matas. 2017. DeblurGAN: Blind Motion Deblurring Using Conditional Adversarial Networks. arXiv preprint arXiv:1711.07064 (2017).Google Scholar
41. Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew P Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. 2017. Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. In CVPR, Vol. 2. 4.Google Scholar
42. Anat Levin, Samuel W Hasinoff, Paul Green, Frédo Durand, and William T Freeman. 2009. 4D frequency analysis of computational cameras for depth of field extension. In ACM Transactions on Graphics (TOG), Vol. 28. ACM, 97.Google Scholar
43. Light.co. 2018. Light L16 Camera. https://light.coGoogle Scholar
44. Daniel Malacara-Hernández and Zacarías Malacara-Hernández. 2016. Handbook of optical design. CRC Press.Google Scholar
45. Kaushik Mitra, Oliver Cossairt, and Ashok Veeraraghavan. 2014. To denoise or deblur: parameter optimization for imaging systems. In Digital Photography X, Vol. 9023. International Society for Optics and Photonics, 90230G.Google Scholar
46. Anish Mittal, Anush Krishna Moorthy, and Alan Conrad Bovik. 2012. No-reference image quality assessment in the spatial domain. IEEE Transactions on Image Processing 21, 12 (2012), 4695–4708.Google ScholarDigital Library
47. MobilEye. 2018. MobileEye TriCam. https://press.zf.com/site/press/en_de/microsites/press/list/release/release_41735.htmlGoogle Scholar
48. Mehjabin Monjur, Leonidas Spinoulas, Patrick R Gill, and David G Stork. 2015. Ultra-miniature, computationally efficient diffractive visual-bar-position sensor. In Proc. SensorComm. IEIFSA.Google Scholar
49. Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. 2017. Deep multi-scale convolutional neural network for dynamic scene deblurring. In CVPR, Vol. 1. 3.Google Scholar
50. Ren Ng, Marc Levoy, Mathieu Brédif, Gene Duval, Mark Horowitz, Pat Hanrahan, et al. 2005. Light field photography with a hand-held plenoptic camera. (2005).Google Scholar
51. Yi-Ren Ng, Patrick M Hanrahan, Mark A Horowitz, and Marc S Levoy. 2012. Correction of optical aberrations. US Patent 8,243,157.Google Scholar
52. Augustus Odena, Vincent Dumoulin, and Chris Olah. 2016. Deconvolution and checkerboard artifacts. Distill 1, 10 (2016), e3.Google ScholarCross Ref
53. Steve Oliver, Rick Lake, Shashikant Hegde, Jeff Viens, and Jacques Duparre. 2010. Imaging module with symmetrical lens system and method of manufacture. US Patent 7,710,667.Google Scholar
54. Marios Papas, Thomas Houit, Derek Nowrouzezahrai, Markus H Gross, and Wojciech Jarosz. 2012. The magic lens: refractive steganography. ACM Trans. Graph. 31, 6 (2012), 186–1.Google ScholarDigital Library
55. Jim R Parker. 2010. Algorithms for image processing and computer vision. John Wiley & Sons.Google Scholar
56. Yifan Peng, Xiong Dun, Qilin Sun, and Wolfgang Heidrich. 2017. Mix-and-match holography. ACM Transactions on Graphics (TOG) 36, 6 (2017), 191.Google ScholarDigital Library
57. Yifan Peng, Qiang Fu, Hadi Amata, Shuochen Su, Felix Heide, and Wolfgang Heidrich. 2015. Computational imaging using lightweight diffractive-refractive optics. Optics Express 23, 24 (2015), 31393–31407.Google ScholarCross Ref
58. Yifan Peng, Qiang Fu, Felix Heide, and Wolfgang Heidrich. 2016. The diffractive achromat full spectrum computational imaging with diffractive optics. ACM Transactions on Graphics (TOG) 35, 4 (2016), 31.Google ScholarDigital Library
59. Rajeev Ramanath, Wesley Snyder, Youngjun Yoo, and Mark Drew. 2005. Color image processing pipeline in digital still cameras. IEEE Signal Processing Magazine 22, 1 (2005), 34–43.Google ScholarCross Ref
60. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention. Springer, 234–241.Google ScholarCross Ref
61. Christian J. Schuler, Harold Christopher Burger, Stefan Harmeling, and Bernhard Scholkopf. 2013. A Machine Learning Approach for Non-blind Image Deconvolution. In Proc. Computer Vision and Pattern Recognition.Google ScholarDigital Library
62. Yuliy Schwartzburg, Romain Testuz, Andrea Tagliasacchi, and Mark Pauly. 2014. High-contrast computational caustic design. ACM Transactions on Graphics (TOG) 33, 4 (2014), 74.Google ScholarDigital Library
63. Yichang Shih, Brian Guenter, and Neel Joshi. 2012. Image enhancement using calibrated lens simulations. In European Conference on Computer Vision. Springer, 42–56.Google ScholarDigital Library
64. Vincent Sitzmann, Steven Diamond, Yifan Peng, Xiong Dun, Stephen Boyd, Wolfgang Heidrich, Felix Heide, and Gordon Wetzstein. 2018. End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging. ACM Transactions on Graphics (TOG) 37, 4 (2018), 114.Google ScholarDigital Library
65. Georgii Georgievich Sliusarev. 1984. Aberration and optical design theory. Bristol, England, Adam Hilger, Ltd., 1984, 672 p. Translation. (1984).Google Scholar
66. Warren J. Smith. 2005. Modern lens design. McGraw-Hill.Google Scholar
67. David G Stork and Patrick R Gill. 2013. Lensless ultra-miniature CMOS computational imagers and sensors. Proc. SENSORCOMM (2013), 186–190.Google Scholar
68. David G Stork and Patrick R Gill. 2014. Optical, mathematical, and computational foundations of lensless ultra-miniature diffractive imagers and sensors. International Journal on Advances in Systems and Measurements 7, 3 (2014), 4.Google Scholar
69. Libin Sun, Sunghyun Cho, Jue Wang, and James Hays. 2013. Edge-based blur kernel estimation using patch priors. In Proc. International Conference on Computational Photography (ICCP). 1–8.Google Scholar
70. Kartik Venkataraman, Dan Lelescu, Jacques Duparré, Andrew McMahon, Gabriel Molina, Priyam Chatterjee, Robert Mullis, and Shree Nayar. 2013. Picam: An ultra-thin high performance monolithic camera array. ACM Transactions on Graphics (TOG) 32, 6 (2013), 166.Google ScholarDigital Library
71. Peng Wang, Nabil Mohammad, and Rajesh Menon. 2016. Chromatic-aberration-corrected diffractive lenses for ultra-broadband focusing. Scientific Reports 6 (2016).Google Scholar
72. Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13, 4 (2004), 600–612.Google ScholarDigital Library
73. Yicheng Wu, Vivek Boominathan, Huaijin Chen, Aswin Sankaranarayanan, and Ashok Veeraraghavan. 2019. PhaseCam3D – Learning Phase Masks for Passive Single View Depth Estimation. In Proc. ICCP.Google ScholarCross Ref
74. Li Xu, Jimmy SJ Ren, Ce Liu, and Jiaya Jia. 2014. Deep convolutional neural network for image deconvolution. In Advances in Neural Information Processing Systems. 1790–1798.Google ScholarDigital Library
75. Xiaoyun Yuan, Lu Fang, Qionghai Dai, David J Brady, and Yebin Liu. 2017. Multiscale gigapixel video: A cross resolution image matching and warping approach. In 2017 IEEE International Conference on Computational Photography (ICCP). IEEE, 1–9.Google ScholarCross Ref
76. Jiawei Zhang, Jinshan Pan, Wei-Sheng Lai, Rynson WH Lau, and Ming-Hsuan Yang. 2017. Learning fully convolutional networks for iterative non-blind deconvolution. (2017).Google Scholar
77. Zhengyou Zhang. 2000. A flexible new technique for camera calibration. IEEE Transactions on pattern analysis and machine intelligence 22 (2000).Google ScholarDigital Library
78. Jun Zhu, Tong Yang, and Guofan Jin. 2013. Design method of surface contour for a freeform lens with wide linear field-of-view. Optics express 21, 22 (2013), 26080–26092.Google Scholar
79. Margarete Zoberbier, Sven Hansen, Marc Hennemeyer, Dietrich Tönnies, Ralph Zoberbier, Markus Brehm, Andreas Kraft, Martin Eisner, and Reinhard Völkel. 2009. Wafer level cameras—novel fabrication and packaging technologies. In Int. Image Sens. Workshop.Google Scholar
