“End-to-end complex lens design with differentiate ray tracing” by Sun, Wang, Fu, Dun and Heidrich

Imaging systems have long been designed in separated steps: experience-driven optical design followed by sophisticated image processing. Although recent advances in computational imaging aim to bridge the gap in an end-to-end fashion, the image formation models used in these approaches have been quite simplistic, built either on simple wave optics models such as Fourier transform, or on similar paraxial models. Such models only support the optimization of a single lens surface, which limits the achievable image quality.To overcome these challenges, we propose a general end-to-end complex lens design framework enabled by a differentiable ray tracing image formation model. Specifically, our model relies on the differentiable ray tracing rendering engine to render optical images in the full field by taking into account all on/off-axis aberrations governed by the theory of geometric optics. Our design pipeline can jointly optimize the lens module and the image reconstruction network for a specific imaging task. We demonstrate the effectiveness of the proposed method on two typical applications, including large field-of-view imaging and extended depth-of-field imaging. Both simulation and experimental results show superior image quality compared with conventional lens designs. Our framework offers a competitive alternative for the design of modern imaging systems.

1. Nick Antipa, Grace Kuo, Reinhard Heckel, Ben Mildenhall, Emrah Bostan, Ren Ng, and Laura Waller. 2018. DiffuserCam: lensless single-exposure 3D imaging. Optica 5, 1 (2018), 1–9.Google ScholarCross Ref
2. Martin Arjovsky, Soumith Chintala, and Léon Bottou. 2017. Wasserstein Generative Adversarial Networks. In Proceedings of the 34th International Conference on Machine Learning – Volume 70 (Sydney, NSW, Australia) (ICML’17). JMLR.org, 214–223.Google Scholar
3. Seung-Hwan Baek, Hayato Ikoma, Daniel S Jeon, Yuqi Li, Wolfgang Heidrich, Gordon Wetzstein, and Min H Kim. 2020. End-to-end hyperspectral-depth imaging with learned diffractive optics. arXiv preprint arXiv:2009.00463 (2020).Google Scholar
4. Sai Bangaru, Tzu-Mao Li, and Frédo Durand. 2020. Unbiased Warped-Area Sampling for Differentiable Rendering. ACM Transactions on Graphics (TOG) 39, 6 (2020), 245:1–245:18.Google ScholarDigital Library
5. V. Boominathan, J. K. Adams, J. T. Robinson, and A. Veeraraghavan. 2020. PhlatCam: Designed Phase-Mask Based Thin Lensless Camera. IEEE Transactions on Pattern Analysis andMachine Intelligence 42, 7 (2020), 1618–1629.Google ScholarCross Ref
6. David J Brady, Michael E Gehm, Ronald A Stack, Daniel L Marks, David S Kittle, Dathon R Golish, EM Vera, and Steven D Feller. 2012. Multiscale gigapixel photography. Nature 486, 7403 (2012), 386.Google ScholarCross Ref
7. W Thomas Cathey and Edward R Dowski. 2002. New paradigm for imaging systems. Applied Optics 41, 29 (2002), 6080–6092.Google ScholarCross Ref
8. Ayan Chakrabarti. 2016. Learning Sensor Multiplexing Design through Back-propagation. In Advances in Neural Information Processing Systems, D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (Eds.), Vol. 29. Curran Associates, Inc.Google Scholar
9. Julie Chang and Gordon Wetzstein. 2019a. Deep Optics for Monocular Depth Estimation and 3D Object Detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).Google ScholarCross Ref
10. Julie Chang and Gordon Wetzstein. 2019b. Deep optics for monocular depth estimation and 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 10193–10202.Google ScholarCross Ref
11. Chen Chen, Qifeng Chen, Jia Xu, and Vladlen Koltun. 2018. Learning to See in the Dark. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2018).Google ScholarCross Ref
12. Shane Colburn, Alan Zhan, and Arka Majumdar. 2018. Metasurface optics for full-color computational imaging. Science Advances 4, 2 (2018).Google Scholar
13. Oliver Cossairt and Shree Nayar. 2010. Spectral focal sweep: Extended depth of field from chromatic aberrations. In IEEE International Conference on Computational Photography (ICCP). IEEE, 1–8.Google ScholarCross Ref
14. O. Cossairt, C. Zhou, and S.K. Nayar. 2010. Diffusion Coding Photography for Extended Depth of Field. ACM Transactions on Graphics (TOG) (Aug 2010).Google Scholar
15. O. S. Cossairt, D. Miau, and S. K. Nayar. 2011. Gigapixel Computational Imaging. In IEEE International Conference on Computational Photography (ICCP). 1–8.Google Scholar
16. Geoffroi Côté, Jean-François Lalonde, and Simon Thibault. 2019. Extrapolating from lens design databases using deep learning. Opt. Express 27, 20 (Sep 2019), 28279–28292.Google ScholarCross Ref
17. Geoffroi Côté, Jean-François Lalonde, and Simon Thibault. 2021. Deep learning-enabled framework for automatic lens design starting point generation. Opt. Express 29, 3 (Feb 2021), 3841–3854.Google ScholarCross Ref
18. Paul E. Debevec and Jitendra Malik. 1997. Recovering High Dynamic Range Radiance Maps from Photographs. In Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’97). ACM Press/Addison-Wesley Publishing Co., USA, 369–378.Google Scholar
19. Edward R Dowski and W Thomas Cathey. 1995. Extended depth of field through wave-front coding. Applied optics 34, 11 (1995), 1859–1866.Google Scholar
20. Xiong Dun, Hayato Ikoma, Gordon Wetzstein, Zhanshan Wang, Xinbin Cheng, and Yifan Peng. 2020. Learned rotationally symmetric diffractive achromat for full-spectrum computational imaging. Optica 7, 8 (Aug 2020), 913–922.Google ScholarCross Ref
21. FZ Fang, XD Zhang, A Weckenmann, GX Zhang, and C Evans. 2013. Manufacturing and measurement of freeform optics. CIRP Annals 62, 2 (2013), 823–846.Google ScholarCross Ref
22. Angel Flores, Michael R. Wang, and Jame J. Yang. 2004. Achromatic hybrid refractive-diffractive lens with extended depth of focus. Applied Optics 43, 30 (Oct 2004), 5618–5630.Google ScholarCross Ref
23. Grant R Fowles. 2012. Introduction to modern optics. Courier Dover Publications.Google Scholar
24. Qi Guo, Iuri Frosio, Orazio Gallo, Todd Zickler, and Jan Kautz. 2018. Tackling 3D ToF Artifacts Through Learning and the FLAT Dataset. In The European Conference on Computer Vision (ECCV). Springer.Google ScholarDigital Library
25. Harel Haim, Shay Elmalem, Raja Giryes, Alex Bronstein, and Emanuel Marom. 2018. Depth Estimation From a Single Image Using Deep Learned Phase Coded Mask. IEEE Transactions on Computational Imaging 4 (2018), 298–310.Google ScholarCross Ref
26. Samuel W Hasinoff and Kiriakos N Kutulakos. 2011. Light-efficient photography. IEEE Transactions on Pattern Analysis and Machine Intelligence 33, 11 (2011), 2203–2214.Google ScholarDigital Library
27. Felix Heide, Qiang Fu, Yifan Peng, and Wolfgang Heidrich. 2016. Encoded diffractive optics for full-spectrum computational imaging. Scientific Reports 6 (2016).Google Scholar
28. Roarke Horstmeyer, Richard Y. Chen, Barbara Kappes, and Benjamin Judkewitz. 2017. Convolutional neural networks that teach microscopes how to image. ArXiv abs/1709.07223 (2017).Google Scholar
29. Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017. Image-to-Image Translation with Conditional Adversarial Networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2017).Google ScholarCross Ref
30. Francis A Jenkins and Harvey E White. 2018. Fundamentals of optics. Tata McGraw-Hill Education.Google Scholar
31. Daniel S. Jeon, Seung-Hwan Baek, Shinyoung Yi, Qiang Fu, Xiong Dun, Wolfgang Heidrich, and Min H. Kim. 2019. Compact Snapshot Hyperspectral Imaging with Diffracted Rotation. ACM Transactions on Graphics (TOG) 38, 4 (2019), 117:1–13.Google ScholarDigital Library
32. Michael Kellman, Emrah Bostan, Michael Chen, and Laura Waller. 2019. Data-Driven Design for Fourier Ptychographic Microscopy. In IEEE International Conference on Computational Photography (ICCP). IEEE, 1–8.Google Scholar
33. Salman S. Khan, Adarsh V. R., Vivek Boominathan, Jasper Tan, Ashok Veeraraghavan, and Kaushik Mitra. 2019. Towards Photorealistic Reconstruction of Highly Multiplexed Lensless Images. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).Google ScholarCross Ref
34. Craig Kolb, Don Mitchell, and Pat Hanrahan. 1995. A realistic camera model for computer graphics. In Proceedings of the 22nd annual conference on Computer graphics and interactive techniques. 317–324.Google ScholarDigital Library
35. Alankar Kotwal, Anat Levin, and Ioannis Gkioulekas. 2020. Interferometric Transmission Probing with Coded Mutual Intensity. 39, 4, Article 74 (July 2020), 16 pages.Google Scholar
36. Orest Kupyn, Volodymyr Budzan, Mykola Mykhailych, Dmytro Mishkin, and Jiri Matas. 2017. DeblurGAN: Blind Motion Deblurring Using Conditional Adversarial Networks. arXiv preprint arXiv:1711.07064 (2017).Google Scholar
37. Orest Kupyn, Tetiana Martyniuk, Junru Wu, and Zhangyang Wang. 2019. DeblurGANv2: Deblurring (Orders-of-Magnitude) Faster and Better. In The IEEE International Conference on Computer Vision (ICCV).Google Scholar
38. Anat Levin. 2010. Analyzing Depth from Coded Aperture Sets. In Computer Vision – ECCV 2010, Kostas Daniilidis, Petros Maragos, and Nikos Paragios (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 214–227.Google ScholarCross Ref
39. Anat Levin, Rob Fergus, Frédo Durand, and William T. Freeman. 2007. Image and Depth from a Conventional Camera with a Coded Aperture. ACM Transactions on Graphics (TOG) 26, 3 (July 2007), 70–es.Google ScholarDigital Library
40. Anat Levin, Samuel W Hasinoff, Paul Green, Frédo Durand, and William T Freeman. 2009. 4D frequency analysis of computational cameras for depth of field extension. In ACM Transactions on Graphics (TOG), Vol. 28. ACM, 97.Google Scholar
41. Zhiqiang Liu, Angel Flores, Michael R. Wang, and Jianwen J. Yang. 2007. Diffractive infrared lens with extended depth of focus. Optical Engineering 46, 1 (2007), 1 — 9.Google ScholarCross Ref
42. Daniel Malacara-HernáUndez and Zacarías Malacara-Hernández. 2016. Handbook of optical design. CRC Press.Google Scholar
43. S. Mann and Rosalind W. Picard. 1994. Being ‘undigital’ with digital cameras: extending dynamic range by combining differently exposed pictures.Google Scholar
44. Christopher A Metzler, Hayato Ikoma, Yifan Peng, and Gordon Wetzstein. 2020. Deep optics for single-shot high-dynamic-range imaging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1375–1385.Google ScholarCross Ref
45. Mehjabin Monjur, Leonidas Spinoulas, Patrick R Gill, and David G Stork. 2015. Ultra-miniature, computationally efficient diffractive visual-bar-position sensor. In Proc. SensorComm. IEIFSA.Google Scholar
46. Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. 2017. Deep multi-scale convolutional neural network for dynamic scene deblurring. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 1, 2, 3.Google ScholarCross Ref
47. S.K. Nayar, V. Branzoi, and T. Boult. 2004. Programmable Imaging using a Digital Micromirror Array. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) I, 436–443.Google Scholar
48. Elias Nehme, Daniel Freedman, Racheli Gordon, Boris Ferdman, Tomer Michaeli, and Yoav Shechtman. 2019. Dense three dimensional localization microscopy by deep learning.Google Scholar
49. Merlin Nimier-David, Delio Vicini, Tizian Zeltner, and Wenzel Jakob. 2019. Mitsuba 2: A Retargetable Forward and Inverse Renderer. ACM Transactions on Graphics (TOG) 38, 6 (Dec. 2019).Google ScholarDigital Library
50. Yifan Peng, Qilin Sun, Xiong Dun, Gordon Wetzstein, Wolfgang Heidrich, and Felix Heide. 2019. Learned Large Field-of-View Imaging with Thin-Plate Optics. ACM Transactions on Graphics (TOG) 38, 6, Article 219 (Nov. 2019), 14 pages.Google ScholarDigital Library
51. E. Reinhard and K. Devlin. 2005. Dynamic range reduction inspired by photoreceptor physiology. IEEE Transactions on Visualization and Computer Graphics 11, 1 (2005), 13–24.Google ScholarDigital Library
52. M. Rouf, R. Mantiuk, W. Heidrich, M. Trentacoste, and C. Lau. 2011. Glare Encoding of High Dynamic Range Images. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
53. Christian J. Schuler, Harold Christopher Burger, Stefan Harmeling, and Bernhard Scholkopf. 2013. A Machine Learning Approach for Non-blind Image Deconvolution. In Proc. Computer Vision and Pattern Recognition.Google ScholarDigital Library
54. Yoav Shechtman, Lucien E Weiss, Adam S. Backer, Maurice Y. Lee, and W E Moerner. 2016. Multicolour localization microscopy by point-spread-function engineering. Nature photonics 10 (2016), 590–594.Google Scholar
55. Yichang Shih, Brian Guenter, and Neel Joshi. 2012. Image enhancement using calibrated lens simulations. In European Conference on Computer Vision (ECCV). Springer, 42–56.Google ScholarDigital Library
56. Vincent Sitzmann, Steven Diamond, Yifan Peng, Xiong Dun, Stephen Boyd, Wolfgang Heidrich, Felix Heide, and Gordon Wetzstein. 2018. End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging. ACM Transactions on Graphics (TOG) 37, 4 (2018), 114.Google ScholarDigital Library
57. Warren J. Smith. 2005. Modern lens design. McGraw-Hill.Google Scholar
58. David G Stork and Patrick R Gill. 2013. Lensless ultra-miniature CMOS computational imagers and sensors. Proc. SENSORCOMM (2013), 186–190.Google Scholar
59. David G Stork and Patrick R Gill. 2014. Optical, mathematical, and computational foundations of lensless ultra-miniature diffractive imagers and sensors. International Journal on Advances in Systems and Measurements 7, 3 (2014), 4.Google Scholar
60. Qilin Sun, Xiong Dun, Yifan Peng, and Wolfgang Heidrich. 2018. Depth and Transient Imaging With Compressive SPAD Array Cameras. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Google ScholarCross Ref
61. Qilin Sun, Ethan Tseng, Qiang Fu, Wolfgang Heidrich, and Felix Heide. 2020a. Learning Rank-1 Diffractive Optics for Single-Shot High Dynamic Range Imaging. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Google ScholarCross Ref
62. Qilin Sun, Jian Zhang, Xiong Dun, Bernard Ghanem, Yifan Peng, and Wolfgang Heidrich. 2020b. End-to-End Learned, Optically Coded Super-Resolution SPAD Camera. ACM Transactions on Graphics (TOG) 39, 2, Article 9 (March 2020), 14 pages.Google ScholarDigital Library
63. Sara C Tucker, W Thomas Cathey, and Edward R Dowski. 1999. Extended depth of field and aberration control for inexpensive digital microscope systems. Optics Express 4, 11 (1999), 467–474.Google ScholarCross Ref
64. Kartik Venkataraman, Dan Lelescu, Jacques Duparré, Andrew McMahon, Gabriel Molina, Priyam Chatterjee, Robert Mullis, and Shree Nayar. 2013. Picam:Anultra-thin high performance monolithic camera array. ACM Transactions on Graphics (TOG) 32, 6 (2013), 166.Google ScholarDigital Library
65. Yicheng Wu, Vivek Boominathan, Huaijin Chen, Aswin Sankaranarayanan, and Ashok Veeraraghavan. 2019a. PhaseCam3D – Learning Phase Masks for Passive Single View Depth Estimation. In IEEE International Conference on Computational Photography (ICCP).Google Scholar
66. Y. Wu, V. Boominathan, H. Chen, A. Sankaranarayanan, and A. Veeraraghavan. 2019b. PhaseCam3D — Learning Phase Masks for Passive Single View Depth Estimation. In IEEE International Conference on Computational Photography (ICCP). IEEE Computer Society, Los Alamitos, CA, USA, 1–12.Google Scholar
67. Y. Wu, F. Li, F. Willomitzer, A. Veeraraghavan, and O. Cossairt. 2020. WISHED: Wavefront imaging sensor with high resolution and depth ranging. In IEEE International Conference on Computational Photography (ICCP). 1–10.Google Scholar
68. Li Xu, Jimmy SJ Ren, Ce Liu, and Jiaya Jia. 2014. Deep convolutional neural network for image deconvolution. In Advances in Neural Information Processing Systems. 1790–1798.Google ScholarDigital Library
69. Xiaoyun Yuan, Lu Fang, Qionghai Dai, David J Brady, and Yebin Liu. 2017. Multiscale gigapixel video: A cross resolution image matching and warping approach. In IEEE International Conference on Computational Photography (ICCP). IEEE, 1–9.Google ScholarCross Ref
70. Cheng Zhang, Bailey Miller, Kai Yan, Ioannis Gkioulekas, and Shuang Zhao. 2020. Path-Space Differentiable Rendering. ACM Transactions on Graphics (TOG) 39, 4 (2020), 143:1–143:19.Google ScholarDigital Library
71. Cheng Zhang, Lifan Wu, Changxi Zheng, Ioannis Gkioulekas, Ravi Ramamoorthi, and Shuang Zhao. 2019. A Differential Theory of Radiative Transfer. ACM Transactions on Graphics (TOG) 38, 6 (2019), 227:1–227:16.Google ScholarDigital Library
72. Jiawei Zhang, Jinshan Pan, Wei-Sheng Lai, Rynson WH Lau, and Ming-Hsuan Yang. 2017. Learning fully convolutional networks for iterative non-blind deconvolution. (2017).Google Scholar
73. Xuaner Zhang, Ren Ng, and Qifeng Chen. 2018. Single Image Reflection Separation with PerceptualLosses. In IEEE Conference on Computer Vision and Pattern Recognition.Google Scholar