“Neural complex luminaires: representation and rendering” by Zhu, Bai, Xu, Bako, Velázquez-Armendáriz, et al. …

Complex luminaires, such as grand chandeliers, can be extremely costly to render because the light-emitting sources are typically encased in complex refractive geometry, creating difficult light paths that require many samples to evaluate with Monte Carlo approaches. Previous work has attempted to speed up this process, but the methods are either inaccurate, require the storage of very large lightfields, and/or do not fit well into modern path-tracing frameworks. Inspired by the success of deep networks, which can model complex relationships robustly and be evaluated efficiently, we propose to use a machine learning framework to compress a complex luminaire’s lightfield into an implicit neural representation. Our approach can easily plug into conventional renderers, as it works with the standard techniques of path tracing and multiple importance sampling (MIS). Our solution is to train three networks to perform the essential operations for evaluating the complex luminaire at a specific point and view direction, importance sampling a point on the luminaire given a shading location, and blending to determine the transparency of luminaire queries to properly composite them with other scene elements. We perform favorably relative to state-of-the-art approaches and render final images that are close to the high-sample-count reference with only a fraction of the computation and storage costs, with no need to store the original luminaire geometry and materials.

1. Ian Ashdown. 1995. Near-Field Photometry: Measuring and Modeling Complex 3-D Light Sources. In ACM SIGGRAPH Course Notes. 1–15.Google Scholar
2. Ian Ashdown and Ron Rykowski. 1998. Making Near-Field Photometry Practical. Journal of the Illuminating Engineering Society of North America 27, 1 (1998), 67–79. Google ScholarCross Ref
3. Steve Bako, Mark Meyer, Tony DeRose, and Pradeep Sen. 2019. Offline Deep Importance Sampling for Monte Carlo Path Tracing. Computer Graphics Forum (2019).Google Scholar
4. Steve Bako, Thijs Vogels, Brian McWilliams, Mark Meyer, Jan Novák, Alex Harvill, Pradeep Sen, Tony DeRose, and Fabrice Rousselle. 2017. Kernel-Predicting Convolutional Networks for Denoising Monte Carlo Renderings. ACM Transactions on Graphics 36, 4 (July 2017).Google ScholarDigital Library
5. Mojtaba Bemana, Karol Myszkowski, Hans-Peter Seidel, and Tobias Ritschel. 2019. Neural View-Interpolation for Sparse LightField Video. arXiv preprint arXiv:1910.13921 (2019).Google Scholar
6. Chakravarty R. A. Chaitanya, Anton Kaplanyan, Christoph Schied, Marco Salvi, Aaron Lefohn, Derek Nowrouzezahrai, and Timo Aila. 2017. Interactive Reconstruction of Noisy Monte Carlo Image Sequences using a Recurrent Autoencoder. ACM Transactions on Graphics (July 2017).Google Scholar
7. Chuo-Ling Chang, Xiaoging Zhu, Prashant Ramanathan, and Bernd Girod. 2003. Inter-View Wavelet Compression of Light Fields with Disparity-Compensated Lifting. In Proc. of VCIP. SPIE, 694–706.Google ScholarCross Ref
8. Chuo-Ling Chang, Xiaoqing Zhu, Prashant Ramanathan, and Bernd Girod. 2006. Light Field Compression Using Disparity-Compensated Lifting and Shape Adaptation. IEEE Transactions on Image Processing 15, 4 (2006), 793–806.Google ScholarDigital Library
9. Wei-Chao Chen, Jean-Yves Bouguet, Michael H. Chu, and Radek Grzeszczuk. 2002. Light Field Mapping: Efficient Representation and Hardware Rendering of Surface Light Fields. ACM Trans. Graph. 21, 3 (2002), 447–456.Google ScholarDigital Library
10. Thomas Davies, Derek Nowrouzezahrai, and Alec Jacobson. 2020. Overfit Neural Networks as a Compact Shape Representation. arXiv:2009.09808 [cs.GR]Google Scholar
11. SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Garnelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, et al. 2018. Neural scene representation and rendering. Science 360, 6394 (2018), 1204–1210.Google Scholar
12. Liangsheng Ge, Beibei Wang, Lu Wang, and Nicolas Holzschuch. 2018. A Compact Representation for Multiple Scattering in Participating Media using Neural Networks. In ACM SIGGRAPH 2018 Talks. Vancouver, Canada, 1–2.Google ScholarDigital Library
13. Iliyan Georgiev, Jaroslav Křivánek, Tomáš Davidovič, and Philipp Slusallek. 2012. Light transport simulation with vertex connection and merging. ACM Trans. Graph. 31, 6 (2012), 192:1–192:10.Google ScholarDigital Library
14. Steven J. Gortler, Radek Grzeszczuk, Richard Szeliski, and Michael F. Cohen. 1996. The Lumigraph. In Proc. of SIGGRAPH. 43–54.Google Scholar
15. Jonathan Granskog, Fabrice Rousselle, Marios Papas, and Jan Novák. 2020. Compositional Neural Scene Representations for Shading Inference. ACM Transactions on Graphics (Proceedings of SIGGRAPH) 39, 4 (July 2020).Google Scholar
16. Toshiya Hachisuka, Shinji Ogaki, and Henrik Wann Jensen. 2008. Progressive Photon Mapping. ACM Trans. Graph. 27, 5 (2008), 130:1–130:8.Google ScholarDigital Library
17. Toshiya Hachisuka, Jacopo Pantaleoni, and Henrik Wann Jensen. 2012. A path space extension for robust light transport simulation. ACM Trans. Graph. 31, 6 (2012), 191:1–191:10.Google ScholarDigital Library
18. Wolfgang Heidrich, Jan Kautz, Philipp Slusallek, and Hans-Peter Seidel. 1998. Canned Lightsources. In Rendering techniques’ 98. 293–300.Google Scholar
19. Wenzel Jakob. 2010. Mitsuba Physically Based Renderer. http://www.mitsubarenderer.orgGoogle Scholar
20. Wenzel Jakob and Steve Marschner. 2012. Manifold Exploration: A Markov Chain Monte Carlo Technique for Rendering Scenes with Difficult Specular Transport. ACM Transactions on Graphics (TOG) 31, 4, Article 58 (2012), 58:1–58:13 pages.Google ScholarDigital Library
21. James T. Kajiya. 1986. The Rendering Equation. SIGGRAPH Comput. Graph. 20, 4 (Aug. 1986), 143–150.Google ScholarDigital Library
22. Nima Khademi Kalantari, Steve Bako, and Pradeep Sen. 2015. A Machine Learning Approach for Filtering Monte Carlo Noise. ACM Transactions on Graphics 34, 4, Article 122 (July 2015), 12 pages.Google ScholarDigital Library
23. Nima Khademi Kalantari, Ting-Chun Wang, and Ravi Ramamoorthi. 2016. Learning-based View Synthesis for Light Field Cameras. ACM Transactions on Graphics 35, 6, Article 193 (Nov. 2016), 10 pages.Google ScholarDigital Library
24. Simon Kallweit, Thomas Müller, Brian McWilliams, Markus Gross, and Jan Novák. 2017. Deep Scattering: Rendering Atmospheric Clouds with Radiance-Predicting Neural Networks. ACM Transactions on Graphics 36, 6, Article 231 (Nov. 2017), 11 pages.Google ScholarDigital Library
25. S. Kniep, S. Häring, and M. Magnor. 2009. Efficient and Accurate Rendering of Complex Light Sources. Computer Graphics Forum 28, 4 (June 2009), 1073–1081.Google ScholarDigital Library
26. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems. 1097–1105.Google Scholar
27. Alexandr Kuznetsov, Miloš Hašan, Zexiang Xu, Ling-Qi Yan, Bruce Walter, Nima Khademi Kalantari, Steve Marschner, and Ravi Ramamoorthi. 2019. Learning Generative Models for Rendering Specular Microgeometry. ACM Transactions on Graphics 38, 6, Article 225 (Nov. 2019), 14 pages.Google ScholarDigital Library
28. Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, and Timo Aila. 2018. Noise2Noise: Learning Image Restoration without Clean Data. In Proceedings of the 35th International Conference on Machine Learning, Vol. 80.Google Scholar
29. Marc Levoy and Pat Hanrahan. 1996. Light Field Rendering. Proc. of SIGGRAPH (1996).Google ScholarDigital Library
30. Heqi Lu, Romain Pacanowski, and Xavier Granier. 2015. Position-Dependent Importance Sampling of Light Field Luminaires. IEEE Transactions on Visualization and Computer Graphics 21, 2 (feb 2015), 241–251.Google ScholarDigital Library
31. Albert Mas, Ignacio Martín, and Gustavo Patow. 2008. Compression and Importance Sampling of Near-Field Light Sources. Computer Graphics Forum 27, 8 (Dec. 2008), 2013–2027. Google ScholarCross Ref
32. Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. 2019. Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines. ACM Transactions on Graphics (2019).Google Scholar
33. Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. 2020. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In ECCV 2020.Google ScholarDigital Library
34. Thomas Müller, Brian McWilliams, Fabrice Rousselle, Markus Gross, and Jan Novák. 2019. Neural importance sampling. ACM Transactions on Graphics 38, 5 (2019).Google ScholarDigital Library
35. Julius Muschaweck. 2011. What’s in a ray set: moving towards a unified ray set format, In Illumination Optics II. Proc. SPIE 8170. Google ScholarCross Ref
36. P.Y. Ngai. 1987. On near-field photometry. Journal of the Illuminating Engineering Society 16, 2 (1987), 129–136.Google ScholarCross Ref
37. NVIDIA. 2021. NVIDIA TensorRT. https://developer.nvidia.com/tensorrtGoogle Scholar
38. Gilles Rainer, Wenzel Jakob, Abhijeet Ghosh, and Tim Weyrich. 2019. Neural BTF Compression and Interpolation. In Computer Graphics Forum, Vol. 38. 235–244.Google ScholarCross Ref
39. Peiran Ren, Yue Dong, Stephen Lin, Xin Tong, and Baining Guo. 2015. Image based relighting using neural networks. ACM Transactions on Graphics 34, 4 (2015), 111.Google ScholarDigital Library
40. Peiran Ren, Jinpeng Wang, Minmin Gong, Stephen Lin, Xin Tong, and Baining Guo. 2013. Global Illumination with Radiance Regression Functions. ACM Transactions on Graphics 32 (July 2013).Google Scholar
41. Todd Saemish, P. Ericson, G. Hauser, E. Gibson, R. Heinisch, C. Loch, and IESNA. 2002. ANSI/IESNA Standard File Format for the Electronic Transfer of Photometric Data and Related Information.Google Scholar
42. Pỳnar Satỳlmỳs, Thomas Bashford-Rogers, Alan Chalmers, and Kurt Debattista. 2017. A machine-learning-driven sky model. IEEE computer graphics and applications 37, 1 (2017), 80–91.Google ScholarDigital Library
43. Eric Veach. 1997. Robust Monte Carlo Methods for Light Transport Simulation. Ph.D. Dissertation. Stanford University.Google ScholarDigital Library
44. Eric Veach and Leonidas J. Guibas. 1997. Metropolis Light Transport. In Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’97). ACM Press/Addison-Wesley Publishing Co., USA, 65–76. Google ScholarDigital Library
45. Edgar Velázquez-Armendáriz, Zhao Dong, Bruce Walter, and Donald P. Greenberg. 2015. Complex Luminaires: Illumination and Appearance Rendering. ACM Transactions on Graphics 34, 3 (May 2015), 26:1–26:15.Google ScholarDigital Library
46. Delio Vicini, Vladlen Koltun, and Wenzel Jakob. 2019. A Learned Shape-Adaptive Subsurface Scattering Model. ACM Transactions on Graphics (Proceedings of SIGGRAPH) 38, 4 (July 2019), 126:1–126:15.Google Scholar
47. Thijs Vogels, Fabrice Rousselle, Brian McWilliams, Gerhard Röthlin, Alex Harvill, David Adler, Mark Meyer, and Jan Novák. 2018. Denoising with Kernel Prediction and Asymmetric Loss Functions. ACM Transactions on Graphics 37, 4, Article 124 (2018), 124:1–124:15 pages.Google ScholarDigital Library
48. Ting-Chun Wang, Jun-Yan Zhu, Nima Khademi Kalantari, Alexei A. Efros, and Ravi Ramamoorthi. 2017. Light Field Video Capture Using a Learning-Based Hybrid Imaging System. ACM Transactions on Graphics (Proceedings of SIGGRAPH) 36, 4 (2017).Google Scholar
49. Zexiang Xu, Kalyan Sunkavalli, Sunil Hadap, and Ravi Ramamoorthi. 2018. Deep image-based relighting from optimal sparse samples. ACM Transactions on Graphics 37, 4 (2018), 126.Google ScholarDigital Library
50. Quan Zheng and Matthias Zwicker. 2019. Learning to importance sample in primary sample space. In Computer Graphics Forum, Vol. 38. Wiley Online Library, 169–179.Google Scholar
51. Károly Zsolnai-Fehér, Peter Wonka, and Michael Wimmer. 2018. Gaussian material synthesis. ACM Transactions on Graphics 37, 4 (2018), 76:1–76:14.Google ScholarDigital Library