“Neural complex luminaires: representation and rendering” by Zhu, Bai, Xu, Bako, Velázquez-Armendáriz, et al. …

  • ©Junqiu Zhu, Yaoyi Bai, Zilin Xu, Steve Bako, Edgar Velázquez-Armendáriz, Lu Wang, Pradeep Sen, Milos Hasan, and Ling-Qi Yan




    Neural complex luminaires: representation and rendering



    Complex luminaires, such as grand chandeliers, can be extremely costly to render because the light-emitting sources are typically encased in complex refractive geometry, creating difficult light paths that require many samples to evaluate with Monte Carlo approaches. Previous work has attempted to speed up this process, but the methods are either inaccurate, require the storage of very large lightfields, and/or do not fit well into modern path-tracing frameworks. Inspired by the success of deep networks, which can model complex relationships robustly and be evaluated efficiently, we propose to use a machine learning framework to compress a complex luminaire’s lightfield into an implicit neural representation. Our approach can easily plug into conventional renderers, as it works with the standard techniques of path tracing and multiple importance sampling (MIS). Our solution is to train three networks to perform the essential operations for evaluating the complex luminaire at a specific point and view direction, importance sampling a point on the luminaire given a shading location, and blending to determine the transparency of luminaire queries to properly composite them with other scene elements. We perform favorably relative to state-of-the-art approaches and render final images that are close to the high-sample-count reference with only a fraction of the computation and storage costs, with no need to store the original luminaire geometry and materials.


    1. Ian Ashdown. 1995. Near-Field Photometry: Measuring and Modeling Complex 3-D Light Sources. In ACM SIGGRAPH Course Notes. 1–15.Google Scholar
    2. Ian Ashdown and Ron Rykowski. 1998. Making Near-Field Photometry Practical. Journal of the Illuminating Engineering Society of North America 27, 1 (1998), 67–79. Google ScholarCross Ref
    3. Steve Bako, Mark Meyer, Tony DeRose, and Pradeep Sen. 2019. Offline Deep Importance Sampling for Monte Carlo Path Tracing. Computer Graphics Forum (2019).Google Scholar
    4. Steve Bako, Thijs Vogels, Brian McWilliams, Mark Meyer, Jan Novák, Alex Harvill, Pradeep Sen, Tony DeRose, and Fabrice Rousselle. 2017. Kernel-Predicting Convolutional Networks for Denoising Monte Carlo Renderings. ACM Transactions on Graphics 36, 4 (July 2017).Google ScholarDigital Library
    5. Mojtaba Bemana, Karol Myszkowski, Hans-Peter Seidel, and Tobias Ritschel. 2019. Neural View-Interpolation for Sparse LightField Video. arXiv preprint arXiv:1910.13921 (2019).Google Scholar
    6. Chakravarty R. A. Chaitanya, Anton Kaplanyan, Christoph Schied, Marco Salvi, Aaron Lefohn, Derek Nowrouzezahrai, and Timo Aila. 2017. Interactive Reconstruction of Noisy Monte Carlo Image Sequences using a Recurrent Autoencoder. ACM Transactions on Graphics (July 2017).Google Scholar
    7. Chuo-Ling Chang, Xiaoging Zhu, Prashant Ramanathan, and Bernd Girod. 2003. Inter-View Wavelet Compression of Light Fields with Disparity-Compensated Lifting. In Proc. of VCIP. SPIE, 694–706.Google ScholarCross Ref
    8. Chuo-Ling Chang, Xiaoqing Zhu, Prashant Ramanathan, and Bernd Girod. 2006. Light Field Compression Using Disparity-Compensated Lifting and Shape Adaptation. IEEE Transactions on Image Processing 15, 4 (2006), 793–806.Google ScholarDigital Library
    9. Wei-Chao Chen, Jean-Yves Bouguet, Michael H. Chu, and Radek Grzeszczuk. 2002. Light Field Mapping: Efficient Representation and Hardware Rendering of Surface Light Fields. ACM Trans. Graph. 21, 3 (2002), 447–456.Google ScholarDigital Library
    10. Thomas Davies, Derek Nowrouzezahrai, and Alec Jacobson. 2020. Overfit Neural Networks as a Compact Shape Representation. arXiv:2009.09808 [cs.GR]Google Scholar
    11. SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Garnelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, et al. 2018. Neural scene representation and rendering. Science 360, 6394 (2018), 1204–1210.Google Scholar
    12. Liangsheng Ge, Beibei Wang, Lu Wang, and Nicolas Holzschuch. 2018. A Compact Representation for Multiple Scattering in Participating Media using Neural Networks. In ACM SIGGRAPH 2018 Talks. Vancouver, Canada, 1–2.Google ScholarDigital Library
    13. Iliyan Georgiev, Jaroslav Křivánek, Tomáš Davidovič, and Philipp Slusallek. 2012. Light transport simulation with vertex connection and merging. ACM Trans. Graph. 31, 6 (2012), 192:1–192:10.Google ScholarDigital Library
    14. Steven J. Gortler, Radek Grzeszczuk, Richard Szeliski, and Michael F. Cohen. 1996. The Lumigraph. In Proc. of SIGGRAPH. 43–54.Google Scholar
    15. Jonathan Granskog, Fabrice Rousselle, Marios Papas, and Jan Novák. 2020. Compositional Neural Scene Representations for Shading Inference. ACM Transactions on Graphics (Proceedings of SIGGRAPH) 39, 4 (July 2020).Google Scholar
    16. Toshiya Hachisuka, Shinji Ogaki, and Henrik Wann Jensen. 2008. Progressive Photon Mapping. ACM Trans. Graph. 27, 5 (2008), 130:1–130:8.Google ScholarDigital Library
    17. Toshiya Hachisuka, Jacopo Pantaleoni, and Henrik Wann Jensen. 2012. A path space extension for robust light transport simulation. ACM Trans. Graph. 31, 6 (2012), 191:1–191:10.Google ScholarDigital Library
    18. Wolfgang Heidrich, Jan Kautz, Philipp Slusallek, and Hans-Peter Seidel. 1998. Canned Lightsources. In Rendering techniques’ 98. 293–300.Google Scholar
    19. Wenzel Jakob. 2010. Mitsuba Physically Based Renderer. http://www.mitsubarenderer.orgGoogle Scholar
    20. Wenzel Jakob and Steve Marschner. 2012. Manifold Exploration: A Markov Chain Monte Carlo Technique for Rendering Scenes with Difficult Specular Transport. ACM Transactions on Graphics (TOG) 31, 4, Article 58 (2012), 58:1–58:13 pages.Google ScholarDigital Library
    21. James T. Kajiya. 1986. The Rendering Equation. SIGGRAPH Comput. Graph. 20, 4 (Aug. 1986), 143–150.Google ScholarDigital Library
    22. Nima Khademi Kalantari, Steve Bako, and Pradeep Sen. 2015. A Machine Learning Approach for Filtering Monte Carlo Noise. ACM Transactions on Graphics 34, 4, Article 122 (July 2015), 12 pages.Google ScholarDigital Library
    23. Nima Khademi Kalantari, Ting-Chun Wang, and Ravi Ramamoorthi. 2016. Learning-based View Synthesis for Light Field Cameras. ACM Transactions on Graphics 35, 6, Article 193 (Nov. 2016), 10 pages.Google ScholarDigital Library
    24. Simon Kallweit, Thomas Müller, Brian McWilliams, Markus Gross, and Jan Novák. 2017. Deep Scattering: Rendering Atmospheric Clouds with Radiance-Predicting Neural Networks. ACM Transactions on Graphics 36, 6, Article 231 (Nov. 2017), 11 pages.Google ScholarDigital Library
    25. S. Kniep, S. Häring, and M. Magnor. 2009. Efficient and Accurate Rendering of Complex Light Sources. Computer Graphics Forum 28, 4 (June 2009), 1073–1081.Google ScholarDigital Library
    26. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems. 1097–1105.Google Scholar
    27. Alexandr Kuznetsov, Miloš Hašan, Zexiang Xu, Ling-Qi Yan, Bruce Walter, Nima Khademi Kalantari, Steve Marschner, and Ravi Ramamoorthi. 2019. Learning Generative Models for Rendering Specular Microgeometry. ACM Transactions on Graphics 38, 6, Article 225 (Nov. 2019), 14 pages.Google ScholarDigital Library
    28. Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, and Timo Aila. 2018. Noise2Noise: Learning Image Restoration without Clean Data. In Proceedings of the 35th International Conference on Machine Learning, Vol. 80.Google Scholar
    29. Marc Levoy and Pat Hanrahan. 1996. Light Field Rendering. Proc. of SIGGRAPH (1996).Google ScholarDigital Library
    30. Heqi Lu, Romain Pacanowski, and Xavier Granier. 2015. Position-Dependent Importance Sampling of Light Field Luminaires. IEEE Transactions on Visualization and Computer Graphics 21, 2 (feb 2015), 241–251.Google ScholarDigital Library
    31. Albert Mas, Ignacio Martín, and Gustavo Patow. 2008. Compression and Importance Sampling of Near-Field Light Sources. Computer Graphics Forum 27, 8 (Dec. 2008), 2013–2027. Google ScholarCross Ref
    32. Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. 2019. Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines. ACM Transactions on Graphics (2019).Google Scholar
    33. Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. 2020. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In ECCV 2020.Google ScholarDigital Library
    34. Thomas Müller, Brian McWilliams, Fabrice Rousselle, Markus Gross, and Jan Novák. 2019. Neural importance sampling. ACM Transactions on Graphics 38, 5 (2019).Google ScholarDigital Library
    35. Julius Muschaweck. 2011. What’s in a ray set: moving towards a unified ray set format, In Illumination Optics II. Proc. SPIE 8170. Google ScholarCross Ref
    36. P.Y. Ngai. 1987. On near-field photometry. Journal of the Illuminating Engineering Society 16, 2 (1987), 129–136.Google ScholarCross Ref
    37. NVIDIA. 2021. NVIDIA TensorRT. https://developer.nvidia.com/tensorrtGoogle Scholar
    38. Gilles Rainer, Wenzel Jakob, Abhijeet Ghosh, and Tim Weyrich. 2019. Neural BTF Compression and Interpolation. In Computer Graphics Forum, Vol. 38. 235–244.Google ScholarCross Ref
    39. Peiran Ren, Yue Dong, Stephen Lin, Xin Tong, and Baining Guo. 2015. Image based relighting using neural networks. ACM Transactions on Graphics 34, 4 (2015), 111.Google ScholarDigital Library
    40. Peiran Ren, Jinpeng Wang, Minmin Gong, Stephen Lin, Xin Tong, and Baining Guo. 2013. Global Illumination with Radiance Regression Functions. ACM Transactions on Graphics 32 (July 2013).Google Scholar
    41. Todd Saemish, P. Ericson, G. Hauser, E. Gibson, R. Heinisch, C. Loch, and IESNA. 2002. ANSI/IESNA Standard File Format for the Electronic Transfer of Photometric Data and Related Information.Google Scholar
    42. Pỳnar Satỳlmỳs, Thomas Bashford-Rogers, Alan Chalmers, and Kurt Debattista. 2017. A machine-learning-driven sky model. IEEE computer graphics and applications 37, 1 (2017), 80–91.Google ScholarDigital Library
    43. Eric Veach. 1997. Robust Monte Carlo Methods for Light Transport Simulation. Ph.D. Dissertation. Stanford University.Google ScholarDigital Library
    44. Eric Veach and Leonidas J. Guibas. 1997. Metropolis Light Transport. In Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’97). ACM Press/Addison-Wesley Publishing Co., USA, 65–76. Google ScholarDigital Library
    45. Edgar Velázquez-Armendáriz, Zhao Dong, Bruce Walter, and Donald P. Greenberg. 2015. Complex Luminaires: Illumination and Appearance Rendering. ACM Transactions on Graphics 34, 3 (May 2015), 26:1–26:15.Google ScholarDigital Library
    46. Delio Vicini, Vladlen Koltun, and Wenzel Jakob. 2019. A Learned Shape-Adaptive Subsurface Scattering Model. ACM Transactions on Graphics (Proceedings of SIGGRAPH) 38, 4 (July 2019), 126:1–126:15.Google Scholar
    47. Thijs Vogels, Fabrice Rousselle, Brian McWilliams, Gerhard Röthlin, Alex Harvill, David Adler, Mark Meyer, and Jan Novák. 2018. Denoising with Kernel Prediction and Asymmetric Loss Functions. ACM Transactions on Graphics 37, 4, Article 124 (2018), 124:1–124:15 pages.Google ScholarDigital Library
    48. Ting-Chun Wang, Jun-Yan Zhu, Nima Khademi Kalantari, Alexei A. Efros, and Ravi Ramamoorthi. 2017. Light Field Video Capture Using a Learning-Based Hybrid Imaging System. ACM Transactions on Graphics (Proceedings of SIGGRAPH) 36, 4 (2017).Google Scholar
    49. Zexiang Xu, Kalyan Sunkavalli, Sunil Hadap, and Ravi Ramamoorthi. 2018. Deep image-based relighting from optimal sparse samples. ACM Transactions on Graphics 37, 4 (2018), 126.Google ScholarDigital Library
    50. Quan Zheng and Matthias Zwicker. 2019. Learning to importance sample in primary sample space. In Computer Graphics Forum, Vol. 38. Wiley Online Library, 169–179.Google Scholar
    51. Károly Zsolnai-Fehér, Peter Wonka, and Michael Wimmer. 2018. Gaussian material synthesis. ACM Transactions on Graphics 37, 4 (2018), 76:1–76:14.Google ScholarDigital Library

ACM Digital Library Publication: