“Differentiable signed distance function rendering” by Vicini, Speierer and Jakob

Physically-based differentiable rendering has recently emerged as an attractive new technique for solving inverse problems that recover complete 3D scene representations from images. The inversion of shape parameters is of particular interest but also poses severe challenges: shapes are intertwined with visibility, whose discontinuous nature introduces severe bias in computed derivatives unless costly precautions are taken. Shape representations like triangle meshes suffer from additional difficulties, since the continuous optimization of mesh parameters cannot introduce topological changes.One common solution to these difficulties entails representing shapes using signed distance functions (SDFs) and gradually adapting their zero level set during optimization. Previous differentiable rendering of SDFs did not fully account for visibility gradients and required the use of mask or silhouette supervision, or discretization into a triangle mesh.In this article, we show how to extend the commonly used sphere tracing algorithm so that it additionally outputs a reparameterization that provides the means to compute accurate shape parameter derivatives. At a high level, this resembles techniques for differentiable mesh rendering, though we show that the SDF representation admits a particularly efficient reparameterization that outperforms prior work. Our experiments demonstrate the reconstruction of (synthetic) objects without complex regularization or priors, using only a per-pixel RGB loss.

1. David Adalsteinsson and James A. Sethian. 1995. A Fast Level Set Method for Propagating Interfaces. J. Comput. Phys. 118, 2 (1995), 269–277.Google ScholarDigital Library
2. David Adalsteinsson and James A. Sethian. 1999. The Fast Construction of Extension Velocities in Level Set Methods. J. Comput. Phys. 148, 1 (1999), 2–22.Google ScholarDigital Library
3. Matan Atzmon, Niv Haim, Lior Yariv, Ofer Israelov, Haggai Maron, and Yaron Lipman. 2019. Controlling neural level sets. In Advances in Neural Information Processing Systems (NeurIPS). 2032–2041.Google Scholar
4. Dejan Azinović, Tzu-Mao Li, Anton Kaplanyan, and Matthias Nießner. 2019. Inverse Path Tracing for Joint Material and Lighting Estimation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
5. Sai Bangaru, Tzu-Mao Li, and Frédo Durand. 2020. Unbiased Warped-Area Sampling for Differentiable Rendering. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 39, 6 (2020), 245:1–245:18.Google Scholar
6. Sai Bangaru, Jesse Michel, Kevin Mu, Gilbert Bernstein, Tzu-Mao Li, and Jonathan Ragan-Kelley. 2021. Systematically Differentiating Parametric Discontinuities. ACM Trans. Graph. (Proc. SIGGRAPH) 40, 107 (2021), 107:1–107:17.Google ScholarDigital Library
7. Brent Burley. 2012. Physically-based shading at Disney. SIGGRAPH Course Notes. Practical physically-based shading in film and game production. (2012), 1–27.Google Scholar
8. Forrester Cole, Kyle Genova, Avneesh Sud, Daniel Vlasic, and Zhoutong Zhang. 2021. Differentiable Surface Rendering via Non-Differentiable Sampling. (2021). arXiv:2108.04886Google Scholar
9. Brian Curless and Marc Levoy. 1996. A Volumetric Method for Building Complex Models from Range Images. In SIGGRAPH Comput. Graph. 303–312.Google Scholar
10. Miles Detrixhe, Frédéric Gibou, and Chohong Min. 2013. A parallel fast sweeping method for the eikonal equation. J. Comput. Phys. 237 (2013), 46–55.Google ScholarDigital Library
11. Akio Doi and Akio Koide. 1991. An Efficient Method of Triangulating Equi-Valued Surfaces by Using Tetrahedral Cells. IEICE Transactions on Information and Systems 1, 74 (1991), 214–224.Google Scholar
12. Pau Gargallo, Emmanuel Prados, and Peter Sturm. 2007. Minimizing the Reprojection Error in Surface Reconstruction from Images. The IEEE International Conference on Computer Vision (ICCV) (2007), 1–8.Google ScholarCross Ref
13. Ioannis Gkioulekas, Anat Levin, and Todd Zickler. 2016. An evaluation of computational imaging techniques for heterogeneous inverse scattering. In European Conference on Computer Vision. Springer, 685–701.Google ScholarCross Ref
14. Ioannis Gkioulekas, Shuang Zhao, Kavita Bala, Todd Zickler, and Anat Levin. 2013. Inverse Volume Rendering with Material Dictionaries. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 32, 6, Article 162 (Nov. 2013), 13 pages.Google Scholar
15. José Gomes and Olivier Faugeras. 2000. Reconciling Distance Functions and Level Sets. Journal of Visual Communication and Image Representation 11, 2 (2000), 209–223.Google ScholarDigital Library
16. John C. Hart. 1996. Sphere tracing: A geometric method for the antialiased ray tracing of implicit surfaces. The Visual Computer 12, 10 (1 Jan. 1996), 527–545.Google Scholar
17. Yue Jiang, Dantong Ji, Zhizhong Han, and Matthias Zwicker. 2020. SDFDiff: Differentiable Rendering of Signed Distance Fields for 3D Shape Optimization. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google ScholarCross Ref
18. James T. Kajiya. 1986. The Rendering Equation. In SIGGRAPH Comput. Graph. 143–150.Google Scholar
19. Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. 2018. Neural 3D Mesh Renderer. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google ScholarCross Ref
20. Benjamin Keinert, Henry Schäfer, Johann Korndörfer, Urs Ganse, and Marc Stamminger. 2014. Enhanced Sphere Tracing. In Smart Tools and Apps for Graphics – Eurographics Italian Chapter Conference. The Eurographics Association.Google Scholar
21. Pramook Khungurn, Daniel Schroeder, Shuang Zhao, Kavita Bala, and Steve Marschner. 2015. Matching Real Fabrics with Micro-Appearance Models. ACM Trans. Graph. 35, 1, Article 1 (Dec. 2015), 26 pages.Google ScholarDigital Library
22. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations (ICLR).Google Scholar
23. Samuli Laine, Janne Hellsten, Tero Karras, Yeongho Seol, Jaakko Lehtinen, and Timo Aila. 2020. Modular Primitives for High-Performance Differentiable Rendering. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 39, 6 (2020).Google Scholar
24. A. Laurentini. 1994. The visual hull concept for silhouette-based image understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence 16, 2 (1994), 150–162.Google ScholarDigital Library
25. Shingyu Leung and Jianliang Qian. 2006. An adjoint state method for three-dimensional transmission traveltime tomography using first-arrivals. Communications in Mathematical Sciences 4, 1 (2006), 249 — 266.Google ScholarCross Ref
26. Chunming Li, Chenyang Xu, Changfeng Gui, and M.D. Fox. 2005. Level set evolution without re-initialization: a new variational formulation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 430–436.Google Scholar
27. Tzu-Mao Li, Miika Aittala, Frédo Durand, and Jaakko Lehtinen. 2018. Differentiable Monte Carlo Ray Tracing through Edge Sampling. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 37, 6 (2018), 222:1–222:11.Google Scholar
28. Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. 2019. Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning. The IEEE International Conference on Computer Vision (ICCV) (Oct. 2019).Google ScholarCross Ref
29. Shaohui Liu, Yinda Zhang, Songyou Peng, Boxin Shi, Marc Pollefeys, and Zhaopeng Cui. 2020. DIST: Rendering Deep Implicit Signed Distance Function with Differentiable Sphere Tracing. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
30. Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. 2019. Neural Volumes: Learning Dynamic Renderable Volumes from Images. ACM Trans. Graph. (Proc. SIGGRAPH) 38, 4, Article 65 (July 2019), 14 pages.Google ScholarDigital Library
31. Matthew M. Loper and Michael J. Black. 2014. OpenDR: An Approximate Differentiable Renderer. In European Conference on Computer Vision (ECCV). Springer, 154–169.Google Scholar
32. William E. Lorensen and Harvey E. Cline. 1987. Marching Cubes: A High Resolution 3D Surface Construction Algorithm. In SIGGRAPH Comput. Graph. 163–169.Google Scholar
33. Guillaume Loubet, Nicolas Holzschuch, and Wenzel Jakob. 2019. Reparameterizing discontinuous integrands for differentiable rendering. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 38, 6 (Dec. 2019).Google ScholarDigital Library
34. Antoine McNamara, Adrien Treuille, Zoran Popović, and Jos Stam. 2004. Fluid control using the adjoint method. In ACM Trans. Graph. (Proc. SIGGRAPH), Vol. 23. ACM, 449–456.Google ScholarDigital Library
35. Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. 2020. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In European Conference on Computer Vision (ECCV).Google ScholarDigital Library
36. Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Mueller, and Sanja Fidler. 2022. Extracting Triangular 3D Models, Materials, and Lighting From Images. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google ScholarCross Ref
37. Ken Museth. 2017. Novel Algorithm for Sparse and Parallel Fast Sweeping: Efficient Computation of Sparse Signed Distance Fields. In ACM SIGGRAPH Talks. Article 74.Google Scholar
38. Richard A. Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J. Davison, Pushmeet Kohi, Jamie Shotton, Steve Hodges, and Andrew Fitzgibbon. 2011. KinectFusion: Real-time dense surface mapping and tracking. In 2011 10th IEEE International Symposium on Mixed and Augmented Reality. 127–136.Google ScholarDigital Library
39. Baptiste Nicolet, Alec Jacobson, and Wenzel Jakob. 2021. Large Steps in Inverse Rendering of Geometry. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 40, 6 (Dec. 2021).Google Scholar
40. Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. 2020. Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google ScholarCross Ref
41. Matthias Nießner, Michael Zollhöfer, Shahram Izadi, and Marc Stamminger. 2013. Real-Time 3D Reconstruction at Scale Using Voxel Hashing. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 32, 6, Article 169 (Nov. 2013), 11 pages.Google Scholar
42. Merlin Nimier-David, Sébastien Speierer, Benoît Ruiz, and Wenzel Jakob. 2020. Radiative Backpropagation: An Adjoint Method for Lightning-Fast Differentiable Rendering. ACM Trans. Graph. (Proc. SIGGRAPH) 39, 4, Article 146 (July 2020), 15 pages.Google ScholarDigital Library
43. Merlin Nimier-David, Delio Vicini, Tizian Zeltner, and Wenzel Jakob. 2019. Mitsuba 2: A Retargetable Forward and Inverse Renderer. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 38, 6 (Nov. 2019), 17 pages.Google ScholarDigital Library
44. Stanley Osher and James A. Sethian. 1988. Fronts propagating with curvature-dependent speed: Algorithms based on Hamilton-Jacobi formulations. J. Comput. Phys. 79, 1 (1988), 12–49.Google ScholarDigital Library
45. Matt Pharr, Wenzel Jakob, and Greg Humphreys. 2016. Physically Based Rendering: From Theory to Implementation (3rd ed.). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA.Google ScholarDigital Library
46. Edoardo Remelli, Artem Lukoianov, Stephan Richter, Benoit Guillard, Timur Bagautdinov, Pierre Baque, and Pascal Fua. 2020. MeshSDF: Differentiable Iso-Surface Extraction. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 33.Google Scholar
47. Osborne Reynolds. 1903. Papers on mechanical and physical subjects: the sub-mechanics of the universe, Vol. 3. The University Press.Google Scholar
48. Darius Rückert, Linus Franke, and Marc Stamminger. 2021. Adop: Approximate differentiable one-pixel point rendering. (2021). arXiv:2110.06635Google Scholar
49. James A. Sethian. 1996. A fast marching level set method for monotonically advancing fronts. Proceedings of the National Academy of Sciences 93, 4 (1996), 1591–1595.Google ScholarCross Ref
50. James A. Sethian. 1999. Fast Marching Methods. SIAM Rev. 41, 2 (1999), 199–235.Google ScholarDigital Library
51. Dario Seyb, Alec Jacobson, Derek Nowrouzezahrai, and Wojciech Jarosz. 2019. Nonlinear sphere tracing for rendering deformed signed distance fields. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 38, 6 (Nov. 2019).Google Scholar
52. Tianchang Shen, Jun Gao, Kangxue Yin, Ming-Yu Liu, and Sanja Fidler. 2021. Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis. In Advances in Neural Information Processing Systems (NeurIPS).Google Scholar
53. Jos Stam and Ryan Schmidt. 2011. On the Velocity of an Implicit Surface. ACM Trans. Graph. 30, 3, Article 21 (May 2011), 7 pages.Google ScholarDigital Library
54. Yen-Hsi Richard Tsai, Li-Tien Cheng, Stanley Osher, Paul Burchard, and Guillermo Sapiro. 2004. Visibility and its dynamics in a PDE based implicit framework. J. Comput. Phys. 199, 1 (2004), 260–290.Google ScholarDigital Library
55. Eric Veach and Leonidas J. Guibas. 1995. Optimally Combining Sampling Techniques for Monte Carlo Rendering. In SIGGRAPH Comput. Graph. Association for Computing Machinery, New York, NY, USA, 419–428.Google Scholar
56. Delio Vicini, Wenzel Jakob, and Anton Kaplanyan. 2021a. A Non-Exponential Transmittance Model for Volumetric Scene Representations. ACM Trans. Graph. (Proc. SIGGRAPH) 40, 4 (Aug. 2021), 136:1–136:16.Google ScholarDigital Library
57. Delio Vicini, Sébastien Speierer, and Wenzel Jakob. 2021b. Path Replay Backpropagation: Differentiating Light Paths using Constant Memory and Linear Time. ACM Trans. Graph. (Proc. SIGGRAPH) 40, 4 (Aug. 2021), 108:1–108:14.Google ScholarDigital Library
58. Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. 2021. NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction. In Advances in Neural Information Processing Systems (NeurIPS).Google Scholar
59. Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar. 2021. Neural Fields in Visual Computing and Beyond. arXiv:2111.11426Google Scholar
60. Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. 2021. Volume Rendering of Neural Implicit Surfaces. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 34.Google Scholar
61. Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. 2020. Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 33.Google Scholar
62. Wang Yifan, Felice Serena, Shihao Wu, Cengiz Öztireli, and Olga Sorkine-Hornung. 2019. Differentiable Surface Splatting for Point-based Geometry Processing. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 38, 6 (2019).Google Scholar
63. Alex Yu, Sara Fridovich-Keil, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. 2022. Plenoxels: Radiance Fields without Neural Networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
64. Tizian Zeltner, Sébastien Speierer, Iliyan Georgiev, and Wenzel Jakob. 2021. Monte Carlo Estimators for Differential Light Transport. ACM Trans. Graph. (Proc. SIGGRAPH) 40, 4 (Aug. 2021), 78:1–78:16.Google ScholarDigital Library
65. Cheng Zhang, Bailey Miller, Kai Yan, Ioannis Gkioulekas, and Shuang Zhao. 2020. Path-Space Differentiable Rendering. ACM Trans. Graph. (Proc. SIGGRAPH) 39, 4, Article 143 (July 2020), 19 pages.Google ScholarDigital Library
66. Cheng Zhang, Lifan Wu, Changxi Zheng, Ioannis Gkioulekas, Ravi Ramamoorthi, and Shuang Zhao. 2019. A Differential Theory of Radiative Transfer. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 38, 6, Article 227 (Nov. 2019), 16 pages.Google Scholar
67. Cheng Zhang, Zihan Yu, and Shuang Zhao. 2021. Path-Space Differentiable Rendering of Participating Media. ACM Trans. Graph. (Proc. SIGGRAPH) 40, 4 (2021), 76:1–76:15.Google ScholarDigital Library
68. Hongkai Zhao. 2004. A fast sweeping method for eikonal equations. Math. Comp. 74 (2004), 603–627. Issue 250.Google ScholarCross Ref
69. Hong-Kai Zhao, S. Osher, and R. Fedkiw. 2001. Fast surface reconstruction using the level set method. In Proceedings IEEE Workshop on Variational and Level Set Methods in Computer Vision. 194–201.Google Scholar
70. Yang Zhou, Lifan Wu, Ravi Ramamoorthi, and Ling-Qi Yan. 2021. Vectorization for Fast, Analytic, and Differentiable Visibility. ACM Trans. Graph. 40, 3, Article 27 (July 2021), 21 pages.Google ScholarDigital Library