“Depth of Field Aware Differentiable Rendering” by Pidhorskyi, Bagautdinov, Ma, Saragih, Schwartz, et al. …
Conference:
Type(s):
Title:
- Depth of Field Aware Differentiable Rendering
Session/Category Title: Differentiable Rendering
Presenter(s)/Author(s):
Abstract:
Cameras with a finite aperture diameter exhibit defocus for scene elements that are not at the focus distance, and have only a limited depth of field within which objects appear acceptably sharp. In this work we address the problem of applying inverse rendering techniques to input data that exhibits such defocus blurring. We present differentiable depth-of-field rendering techniques that are applicable to both rasterization-based methods using mesh representations, as well as ray-marching-based methods using either explicit [Yu et al. 2021] or implicit volumetric radiance fields [Mildenhall et al. 2020]. Our approach learns significantly sharper scene reconstructions on data containing blur due to depth of field, and recovers aperture and focus distance parameters that result in plausible forward-rendered images. We show applications to macro photography, where typical lens configurations result in a very narrow depth of field, and to multi-camera video capture, where maintaining sharp focus across a large capture volume for a moving subject is difficult.
References:
1. Benjamin Attal, Selena Ling, Aaron Gokaslan, Christian Richardt, and James Tompkin. 2020. MatryODShka: Real-time 6DoF video view synthesis using multi-sphere images. In European Conference on Computer Vision. Springer, 441–459.
2. GR Ayers and J Christopher Dainty. 1988. Iterative blind deconvolution method and its applications. Optics letters 13, 7 (1988), 547–549.
3. Dejan Azinovic, Tzu-Mao Li, Anton Kaplanyan, and Matthias Nießner. 2019. Inverse path tracing for joint material and lighting estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2447–2456.
4. Timur Bagautdinov, Chenglei Wu, Tomas Simon, Fabian Prada, Takaaki Shiratori, Shih-En Wei, Weipeng Xu, Yaser Sheikh, and Jason Saragih. 2021. Driving-signal aware full-body avatars. ACM Transactions on Graphics (TOG) 40, 4 (2021), 1–17.
5. Sai Praveen Bangaru, Tzu-Mao Li, and Frédo Durand. 2020. Unbiased warped-area sampling for differentiable rendering. ACM Transactions on Graphics (TOG) 39, 6 (2020), 1–18.
6. Mario Bertero, Patrizia Boccacci, Gabriele Desiderà, and Giuseppe Vicidomini. 2009. Image deblurring with Poisson data: from cells to galaxies. Inverse Problems 25, 12 (2009), 123006.
7. Jan Biemond, Reginald L Lagendijk, and Russell M Mersereau. 1990. Iterative methods for image deblurring. Proc. IEEE 78, 5 (1990), 856–883.
8. Michael Broxton, John Flynn, Ryan Overbeck, Daniel Erickson, Peter Hedman, Matthew Duvall, Jason Dourgarian, Jay Busch, Matt Whalen, and Paul Debevec. 2020. Immersive light field video with a layered mesh representation. ACM Transactions on Graphics (TOG) 39, 4 (2020), 86–1.
9. Tony F Chan and Chiu-Kwong Wong. 1998. Total variation blind deconvolution. IEEE transactions on Image Processing 7, 3 (1998), 370–375.
10. Blender Online Community. 2018. Blender – a 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam. http://www.blender.org
11. Robert L Cook, Thomas Porter, and Loren Carpenter. 1984. Distributed ray tracing. In Proceedings of the 11th annual conference on Computer graphics and interactive techniques. 137–145.
12. Paul E Debevec and Jitendra Malik. 2008. Recovering high dynamic range radiance maps from photographs. In ACM SIGGRAPH 2008 classes. 1–10.
13. Krzysztof Dudkiewicz. 1995. Real-time depth of field algorithm. In Image Processing for Broadcast and Video Production. Springer, 257–268.
14. Carsten Griwodz, Simone Gasparini, Lilian Calvet, Pierre Gurdjos, Fabien Castan, Benoit Maujean, Gregoire De Lillo, and Yann Lanthony. 2021. AliceVision Meshroom: An open-source 3D reconstruction pipeline. In Proceedings of the 12th ACM Multimedia Systems Conference – MMSys ’21. ACM Press.
15. Paul Haeberli and Kurt Akeley. 1990. The accumulation buffer: hardware support for high-quality rendering. Proceedings of the 17th annual conference on Computer graphics and interactive techniques (1990).
16. Johannes Hanika and Carsten Dachsbacher. 2014. Efficient Monte Carlo Rendering with Realistic Lenses. Comput. Graph. Forum 33, 2 (may 2014), 323–332.
17. Peter Hedman, Pratul P Srinivasan, Ben Mildenhall, Jonathan T Barron, and Paul Debevec. 2021. Baking neural radiance fields for real-time view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 5875–5884.
18. Sébastien Hillaire, Anatole Lécuyer, Rémi Cozot, and Géry Casiez. 2008. Depth-of-field blur effects for first-person navigation in virtual environments. IEEE computer graphics and applications 28, 6 (2008), 47–55.
19. Michael Kass, Aaron Lefohn, and John D Owens. 2006. Interactive depth of field using simulated diffusion on a GPU. (2006).
20. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
21. Craig Kolb, Don Mitchell, and Pat Hanrahan. 1995. A realistic camera model for computer graphics. In Proceedings of the 22nd annual conference on computer graphics and interactive techniques. 317–324.
22. Martin Kraus and Magnus Strengert. 2007. Depth-of-field rendering by pyramidal image processing. In Computer graphics forum, Vol. 26. Wiley Online Library, 645–654.
23. Jaroslav Krivánek, Jiri Zara, and Kadi Bouatouch. 2003. Fast depth of field rendering with surface splatting. In Proceedings Computer Graphics International 2003. IEEE, 196–201.
24. Samuli Laine, Janne Hellsten, Tero Karras, Yeongho Seol, Jaakko Lehtinen, and Timo Aila. 2020. Modular primitives for high-performance differentiable rendering. ACM Transactions on Graphics (TOG) 39, 6 (2020), 1–14.
25. Thomas Leimkühler, Hans-Peter Seidel, and Tobias Ritschel. 2018. Laplacian kernel splatting for efficient depth-of-field and motion blur synthesis or reconstruction. ACM Transactions on Graphics (TOG) 37, 4 (2018), 1–11.
26. Anat Levin, Yair Weiss, Fredo Durand, and William T Freeman. 2009. Understanding and evaluating blind deconvolution algorithms. In 2009 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 1964–1971.
27. Tzu-Mao Li, Miika Aittala, Frédo Durand, and Jaakko Lehtinen. 2018. Differentiable Monte Carlo Ray Tracing through Edge Sampling. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 37, 6 (2018), 222:1–222:11.
28. Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. 2019. Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning. The IEEE International Conference on Computer Vision (ICCV) (Oct 2019).
29. Guillaume Loubet, Nicolas Holzschuch, and Wenzel Jakob. 2019. Reparameterizing discontinuous integrands for differentiable rendering. Transactions on Graphics (Proceedings of SIGGRAPH Asia) 38, 6 (Dec. 2019).
30. Li Ma, Xiaoyu Li, Jing Liao, Qi Zhang, Xuan Wang, Jue Wang, and Pedro V. Sander. 2021a. Deblur-NeRF: Neural Radiance Fields from Blurry Images. CoRR abs/2111.14292 (2021). arXiv:2111.14292 https://arxiv.org/abs/2111.14292
31. Shugao Ma, Tomas Simon, Jason M. Saragih, Dawei Wang, Yuecheng Li, Fernando De la Torre, and Yaser Sheikh. 2021b. Pixel Codec Avatars. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021), 64–73.
32. Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. 2020. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In ECCV.
33. Yuji Moroto, Toshiya Hachisuka, and Nobuyuki Umetani. 2021. Fast Polygonal Splatting using Directional Kernel Difference. (2021).
34. Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. 2022. Instant Neural Graphics Primitives with a Multiresolution Hash Encoding. arXiv preprint arXiv:2201.05989 (2022).
35. Baptiste Nicolet, Alec Jacobson, and Wenzel Jakob. 2021. Large steps in inverse rendering of geometry. ACM Transactions on Graphics (TOG) 40, 6 (2021), 1–13.
36. Merlin Nimier-David, Sébastien Speierer, Benoît Ruiz, and Wenzel Jakob. 2020a. Radiative Backpropagation: An Adjoint Method for Lightning-Fast Differentiable Rendering. Transactions on Graphics (Proceedings of SIGGRAPH) 39, 4 (July 2020).
37. Merlin Nimier-David, Sébastien Speierer, Benoît Ruiz, and Wenzel Jakob. 2020b. Radiative backpropagation: an adjoint method for lightning-fast differentiable rendering. ACM Transactions on Graphics (TOG) 39, 4 (2020), 146–1.
38. Merlin Nimier-David, Delio Vicini, Tizian Zeltner, and Wenzel Jakob. 2019. Mitsuba 2: A Retargetable Forward and Inverse Renderer. Transactions on Graphics (Proceedings of SIGGRAPH Asia) 38, 6 (Dec. 2019).
39. Michael Potmesil and Indranil Chakravarty. 1981. A lens and aperture camera model for synthetic image generation. ACM SIGGRAPH Computer Graphics 15, 3 (1981), 297–305.
40. Peter Quinn, Jerome Parent-Levesque, Cengiz Oztireli, and Derek Nowrouzezahrai. 2020. Differentiable Path Tracing by Regularizing Discontinuities. NeurIPS 2020 Workshop for Differentiable Computer Vision, Graphics and Physics (2020).
41. Przemyslaw Rokita. 1993. Fast generation of depth of field effects in computer graphics. Computers & Graphics 17, 5 (1993), 593–595.
42. Przemyslaw Rokita. 1996. Generating depth of-field effects in virtual reality applications. IEEE Computer Graphics and Applications 16, 2 (1996), 18–21.
43. B. Steinert, H. Dammertz, J. Hanika, and H. P. A. Lensch. 2011. General Spectral Camera Lens Simulation. Computer Graphics Forum (2011).
44. Qilin Sun, Congli Wang, Fu Qiang, Dun Xiong, and Heidrich Wolfgang. 2021. End-to-End Complex Lens Design with Differentiable Ray Tracing. ACM Transactions on Graphics (TOG) 40, 4 (2021).
45. Xin Tao, Hongyun Gao, Xiaoyong Shen, Jue Wang, and Jiaya Jia. 2018. Scale-recurrent network for deep image deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8174–8182.
46. Justus Thies, Michael Zollhöfer, and Matthias Nießner. 2019. Deferred neural rendering: Image synthesis using neural textures. ACM Transactions on Graphics (TOG) 38, 4 (2019), 1–12.
47. Delio Vicini, Sébastien Speierer, and Wenzel Jakob. 2021. Path Replay Backpropagation: Differentiating Light Paths using Constant Memory and Linear Time. ACM Transactions on Graphics (Proceedings of SIGGRAPH) 40, 4 (Aug. 2021), 108:1–108:14.
48. Ruxin Wang and Dacheng Tao. 2014. Recent progress in image deblurring. arXiv preprint arXiv:1409.6838 (2014).
49. Yinhuai Wang, Shuzhou Yang, Yujie Hu, and Jian Zhang. 2022. NeRFocus: Neural Radiance Field for 3D Synthetic Defocus. arXiv preprint arXiv:2203.05189 (2022).
50. Alex Yu, Sara Fridovich-Keil, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. 2021. Plenoxels: Radiance Fields without Neural Networks. arXiv:2112.05131 [cs.CV]
51. Lu Yuan, Jian Sun, Long Quan, and Heung-Yeung Shum. 2007. Image deblurring with blurred/noisy image pairs. In ACM SIGGRAPH 2007 papers. 1–es.
52. Cheng Zhang, Bailey Miller, Kai Yan, Ioannis Gkioulekas, and Shuang Zhao. 2020a. Path-Space Differentiable Rendering. ACM Trans. Graph. 39, 4, Article 143 (jul 2020), 19 pages.
53. Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. 2020b. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492 (2020).
54. Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 586–595.
55. Shuang Zhao, Wenzel Jakob, and Tzu-Mao Li. 2020. Physics-Based Differentiable Rendering: From Theory to Implementation. In ACM SIGGRAPH 2020 Courses (Virtual Event, USA) (SIGGRAPH ’20). Association for Computing Machinery, New York, NY, USA, Article 14, 30 pages.
56. Tianshu Zhou, Jim X Chen, and Mark Pullen. 2007. Accurate depth of field simulation in real time. In Computer Graphics Forum, Vol. 26. Wiley Online Library, 15–23.
