“Modular primitives for high-performance differentiable rendering” by Laine, Hellsten, Karras, Seol, Lehtinen, et al. …
Conference:
Type(s):
Title:
- Modular primitives for high-performance differentiable rendering
Session/Category Title: Differentiable Graphics
Presenter(s)/Author(s):
Abstract:
We present a modular differentiable renderer design that yields performance superior to previous methods by leveraging existing, highly optimized hardware graphics pipelines. Our design supports all crucial operations in a modern graphics pipeline: rasterizing large numbers of triangles, attribute interpolation, filtered texture lookups, as well as user-programmable shading and geometry processing, all in high resolutions. Our modular primitives allow custom, high-performance graphics pipelines to be built directly within automatic differentiation frameworks such as PyTorch or TensorFlow. As a motivating application, we formulate facial performance capture as an inverse rendering problem and show that it can be solved efficiently using our tools. Our results indicate that this simple and straightforward approach achieves excellent geometric correspondence between rendered results and reference imagery.
References:
1. Agisoft. 2020. Agisoft Metashape. https://www.agisoft.com/Google Scholar
2. Oleg Alexander, Mike Rogers, William Lambeth, Matt Chiang, and Paul Debevec. 2009. The Digital Emily Project: Photoreal Facial Modeling and Animation. In ACM SIGGRAPH 2009 Courses (SIGGRAPH ’09).Google ScholarDigital Library
3. Sanjeev Arora, Nadav Cohen, and Elad Hazan. 2018. On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization. In ICML (Proceedings of Machine Learning Research, Vol. 80). 244–253.Google Scholar
4. Thabo Beeler, Bernd Bickel, Paul Beardsley, Bob Sumner, and Markus Gross. 2010. High-Quality Single-Shot Capture of Facial Geometry. ACM Trans. Graph. 29, 4 (2010).Google ScholarDigital Library
5. Thabo Beeler, Fabian Hahn, Derek Bradley, Bernd Bickel, Paul Beardsley, Craig Gotsman, Robert W. Sumner, and Markus Gross. 2011. High-Quality Passive Facial Performance Capture Using Anchor Frames. ACM Trans. Graph. 30, 4 (2011).Google ScholarDigital Library
6. Volker Blanz and Thomas Vetter. 1999. A Morphable Model for the Synthesis of 3D Faces (SIGGRAPH ’99). 187–194.Google Scholar
7. Derek Bradley, Wolfgang Heidrich, Tiberiu Popa, and Alla Sheffer. 2010. High Resolution Passive Facial Performance Capture. ACM Trans. Graph. 29, 4 (2010).Google ScholarDigital Library
8. Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. 2015. ShapeNet: An Information-Rich 3D Model Repository. Technical Report arXiv:1512.03012 [cs.GR]. Stanford University — Princeton University — Toyota Technological Institute at Chicago.Google Scholar
9. Wenzheng Chen, Jun Gao, Huan Ling, Edward Smith, Jaakko Lehtinen, Alec Jacobson, and Sanja Fidler. 2019. Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer. In Advances In Neural Information Processing Systems.Google Scholar
10. R. L. Cook and K. E. Torrance. 1982. A Reflectance Model for Computer Graphics. ACM Trans. Graph. 1, 1 (1982), 7–24.Google ScholarDigital Library
11. Martin de La Gorce, David J. Fleet, and Nikos Paragios. 2011. Model-Based 3D Hand Pose Estimation from Monocular Video. IEEE Transactions on Pattern Analysis and Machine Intelligence 33, 9 (2011), 1793–1805.Google ScholarDigital Library
12. Michael Deering, Stephanie Winner, Bic Schediwy, Chris Duffy, and Neil Hunt. 1988. The Triangle Processor and Normal Vector Shader: A VLSI System for High Performance Graphics. In SIGGRAPH ’88. 21–30.Google ScholarDigital Library
13. DI4D. 2020. DI4D PRO System. https://www.di4d.com/di4d-pro/Google Scholar
14. Ned Greene. 1986. Environment Mapping and Other Applications of World Projections. IEEE Computer Graphics and Applications 6, 11 (1986), 21–29.Google ScholarDigital Library
15. Sebastian Herholz, Oskar Elek, Jiří Vorba, Hendrik Lensch, and Jaroslav Křivánek. 2016. Product Importance Sampling for Light Transport Path Guiding. Computer Graphics Forum 35, 4 (2016), 67–77.Google ScholarDigital Library
16. André Jalobeanu, Frank O. Kuehnel, and John C. Stutz. 2004. Modeling Images of Natural 3D Surfaces: Overview and Potential Applications. 2004 Conference on Computer Vision and Pattern Recognition Workshop (2004).Google ScholarCross Ref
17. Jorge Jimenez, Diego Gutierrez, Jason Yang, Alexander Reshetov, Pete Demoreuille, Tobias Berghoff, Cedric Perthuis, Henry Yu, Morgan McGuire, Timothy Lottes, Hugh Malan, Emil Persson, Dmitry Andreev, and Tiago Sousa. 2011. Filtering Approaches for Real-Time Anti-Aliasing. In ACM SIGGRAPH Courses.Google Scholar
18. Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. 2018. Neural 3D Mesh Renderer. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google ScholarCross Ref
19. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In ICLR.Google Scholar
20. Tzu-Mao Li, Miika Aittala, Frédo Durand, and Jaakko Lehtinen. 2018. Differentiable Monte Carlo Ray Tracing through Edge Sampling. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 37, 6 (2018), 222:1–222:11.Google Scholar
21. Yaron Lipman, Olga Sorkine, David Levin, and Daniel Cohen-Or. 2005. Linear RotationInvariant Coordinates for Meshes. ACM Trans. Graph. 24, 3 (2005), 479–487.Google ScholarDigital Library
22. Guilin Liu, Duygu Ceylan, Ersin Yumer, Jimei Yang, and Jyh-Ming Lien. 2017. Material Editing Using a Physically Based Rendering Network. ICCV (2017), 2280–2288.Google Scholar
23. Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. 2019. Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning. In ICCV.Google Scholar
24. Matthew M. Loper and Michael J. Black. 2014. OpenDR: An Approximate Differentiable Renderer. In ECCV 2014, Vol. 8695. 154–169.Google Scholar
25. Guillaume Loubet, Nicolas Holzschuch, and Wenzel Jakob. 2019. Reparameterizing Discontinuous Integrands for Differentiable Rendering. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 38, 6 (2019).Google Scholar
26. Hugh Malan. 2010. Edge Anti-aliasing by Post-Processing. In GPU Pro, Wolfgang Engel (Ed.). A K Peters, 265–289.Google Scholar
27. Steve Marschner, Stephen H. Westin, Eric P. Lafortune, Kenneth E. Torrance, and Donald P. Greenberg. 1999. Image-Based BRDF Measurement Including Human Skin. In Rendering Techniques.Google ScholarDigital Library
28. Merlin Nimier-David, Delio Vicini, Tizian Zeltner, and Wenzel Jakob. 2019. Mitsuba 2: A Retargetable Forward and Inverse Renderer. ACM Trans. Graph. 38, 6 (2019).Google ScholarDigital Library
29. NVIDIA. 2018. NVIDIA Turing GPU Architecture. Technical Report.Google Scholar
30. Gustavo Patow and Xavier Pueyo. 2003. A Survey of Inverse Rendering Problems. Computer Graphics Forum 22, 4 (2003), 663–687.Google ScholarCross Ref
31. Emil Persson. 2011. Geometric Post-Process Anti-Aliasing. http://www.humus.name/index.php?page=3D&ID=86Google Scholar
32. Bui Tuong Phong. 1975. Illumination for Computer Generated Pictures. Commun. ACM 18, 6 (1975), 311–317.Google ScholarDigital Library
33. R3DS. 2020. R3DS Wrap. https://www.russian3dscanner.com/Google Scholar
34. Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Lo, Justin Johnson, and Georgia Gkioxari. 2020. PyTorch3D. https://github.com/facebookresearch/pytorch3dGoogle Scholar
35. Andreas Schilling, Günter Knittel, and Wolfgang Strasser. 1996. Texram: A Smart Memory for Texturing. IEEE Computer Graphics and Applications 16, 3 (1996), 32–41.Google ScholarDigital Library
36. Ken Shoemake. 1985. Animating Rotation with Quaternion Curves. SIGGRAPH Comput. Graph. 19, 3 (1985), 245–254.Google ScholarDigital Library
37. Olga Sorkine. 2005. Laplacian Mesh Processing. In Eurographics 2005 – State of the Art Reports.Google Scholar
