“LightFormer: Light-oriented Global Neural Rendering in Dynamic Scene”
Conference:
Type(s):
Title:
- LightFormer: Light-oriented Global Neural Rendering in Dynamic Scene
Presenter(s)/Author(s):
Abstract:
The generation of global illumination in real-time has been a long-standing challenge in the graphics community. This work presents a neural rendering approach inspired by many-lights methods and Transformer model, dubbed LightFormer, that can generate realistic global illumination for fully dynamic scenes, including dynamic lighting, materials, cameras, and animated objects.
References:
[1]
Attila T. ?fra. 2024. Intel? Open Image Denoise. https://www.openimagedenoise.org.
[2]
Pontus Andersson, Jim Nilsson, Peter Shirley, and Tomas Akenine-M?ller. 2021. Visualizing errors in rendered high dynamic range images. (2021).
[3]
Steve Bako, Thijs Vogels, Brian McWilliams, Mark Meyer, Jan Nov?k, Alex Harvill, Pradeep Sen, Tony Derose, and Fabrice Rousselle. 2017. Kernel-predicting convolutional networks for denoising Monte Carlo renderings. ACM Trans. Graph. 36, 4 (2017), 97–1.
[4]
Martin Balint, Krzysztof Wolski, Karol Myszkowski, Hans-Peter Seidel, and Rafa? Mantiuk. 2023. Neural Partitioning Pyramids for Denoising Monte Carlo Renderings. In ACM SIGGRAPH 2023 Conference Proceedings. 1–11.
[5]
Aner Ben-Artzi, Kevin Egan, Fr?do Durand, and Ravi Ramamoorthi. 2008. A precomputed polynomial representation for interactive BRDF editing with global illumination. ACM Transactions on Graphics (TOG) 27, 2 (2008), 1–13.
[6]
Aner Ben-Artzi, Ryan Overbeck, and Ravi Ramamoorthi. 2006. Real-time BRDF editing in complex lighting. ACM Transactions on Graphics (TOG) 25, 3 (2006), 945–954.
[7]
James F Blinn. 1977. Models of light reflection for computer synthesized pictures. In Proceedings of the 4th annual conference on Computer graphics and interactive techniques. 192–198.
[8]
Carsten Dachsbacher, Jaroslav K?iv?nek, Milo? Ha?an, Adam Arbree, Bruce Walter, and Jan Nov?k. 2014. Scalable realistic rendering with many-light methods. In Computer Graphics Forum, Vol. 33. Wiley Online Library, 88–104.
[9]
Carsten Dachsbacher and Marc Stamminger. 2005. Reflective shadow maps. In Proceedings of the 2005 symposium on Interactive 3D graphics and games. 203–231.
[10]
Carsten Dachsbacher and Marc Stamminger. 2006. Splatting indirect illumination. In Proceedings of the 2006 symposium on Interactive 3D graphics and games. 93–100.
[11]
Sayantan Datta, Derek Nowrouzezahrai, Christoph Schied, and Zhao Dong. 2022. Neural Shadow Mapping. In ACM SIGGRAPH 2022 Conference Proceedings. 1–9.
[12]
Stavros Diolatzis, Julien Philip, and George Drettakis. 2022. Active exploration for neural global illumination of variable scenes. ACM Transactions on Graphics (TOG) 41, 5 (2022), 1–18.
[13]
Honghao Dong, Guoping Wang, and Sheng Li. 2023. Neural Parametric Mixtures for Path Guiding. In ACM SIGGRAPH 2023 Conference Proceedings. 1–10.
[14]
SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Garnelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, et al. 2018. Neural scene representation and rendering. Science 360, 6394 (2018), 1204–1210.
[15]
Hangming Fan, Rui Wang, Yuchi Huo, and Hujun Bao. 2021. Real-time Monte Carlo Denoising with Weight Sharing Kernel Prediction Network. In Computer Graphics Forum, Vol. 40. Wiley Online Library, 15–27.
[16]
Duan Gao, Haoyuan Mu, and Kun Xu. 2022. Neural Global Illumination: Interactive Indirect Illumination Prediction under Dynamic Area Lights. IEEE Transactions on Visualization and Computer Graphics (2022).
[17]
Jonathan Granskog, Fabrice Rousselle, Marios Papas, and Jan Nov?k. 2020. Compositional neural scene representations for shading inference. ACM Transactions on Graphics (TOG) 39, 4 (2020), 135–1.
[18]
Jonathan Granskog, Till N Schnabel, Fabrice Rousselle, and Jan Nov?k. 2021. Neural scene graph rendering. ACM Transactions on Graphics (TOG) 40, 4 (2021), 1–11.
[19]
Gene Greger, Peter Shirley, Philip M Hubbard, and Donald P Greenberg. 1998. The irradiance volume. IEEE Computer Graphics and Applications 18, 2 (1998), 32–43.
[20]
Jie Guo, Zijing Zong, Yadong Song, Xihao Fu, Chengzhi Tao, Yanwen Guo, and Ling-Qi Yan. 2022. Efficient Light Probes for Real-Time Global Illumination. ACM Transactions on Graphics (TOG) 41, 6 (2022), 1–14.
[21]
Saeed Hadadan, Shuhong Chen, and Matthias Zwicker. 2021. Neural radiosity. ACM Transactions on Graphics (TOG) 40, 6 (2021), 1–11.
[22]
Takahiro Harada, Jay McKee, and Jason C Yang. 2012. Forward+: Bringing deferred lighting to the next level. In Eurographics (Short Papers). 5–8.
[23]
Milo? Ha?an, Fabio Pellacini, and Kavita Bala. 2007. Matrix row-column sampling for the many-light problem. In ACM SIGGRAPH 2007 papers. 26–es.
[24]
Jon Hasselgren, Jacob Munkberg, Marco Salvi, Anjul Patney, and Aaron Lefohn. 2020. Neural temporal adaptive sampling and denoising. In Computer Graphics Forum, Vol. 39. Wiley Online Library, 147–155.
[25]
Yuchi Huo, Rui Wang, Ruzahng Zheng, Hualin Xu, Hujun Bao, and Sung-Eui Yoon. 2020. Adaptive incident radiance field sampling and reconstruction using deep reinforcement learning. ACM Transactions on Graphics (TOG) 39, 1 (2020), 1–17.
[26]
Yuchi Huo and Sung-eui Yoon. 2021. A survey on deep learning-based Monte Carlo denoising. Computational visual media 7 (2021), 169–185.
[27]
Simon Kallweit, Petrik Clarberg, Craig Kolb, Tom’a? Davidovi?, Kai-Hwa Yao, Theresa Foley, Yong He, Lifan Wu, Lucy Chen, Tomas Akenine-M?ller, Chris Wyman, Cyril Crassin, and Nir Benty. 2022. The Falcor Rendering Framework. https://github.com/NVIDIAGameWorks/Falcor
[28]
Anders Wang Kristensen, Tomas Akenine-M?ller, and Henrik Wann Jensen. 2005. Precomputed local radiance transfer for real-time lighting design. In ACM SIGGRAPH 2005 Papers. 1208–1215.
[29]
Alexandr Kuznetsov. 2021. NeuMIP: Multi-resolution neural materials. ACM Transactions on Graphics (TOG) 40, 4 (2021).
[30]
Morgan McGuire and Michael Mara. 2014. Efficient GPU screen-space ray tracing. Journal of Computer Graphics Techniques (JCGT) 3, 4 (2014), 73–85.
[31]
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. 2021. Nerf: Representing scenes as neural radiance fields for view synthesis. Commun. ACM 65, 1 (2021), 99–106.
[32]
Thomas M?ller, Alex Evans, Christoph Schied, and Alexander Keller. 2022. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics (ToG) 41, 4 (2022), 1–15.
[33]
Thomas M?ller, Fabrice Rousselle, Jan Nov?k, and Alexander Keller. 2021. Real-time neural radiance caching for path tracing. arXiv preprint arXiv:2106.12372 (2021).
[34]
Oliver Nalbach, Elena Arabadzhiyska, Dushyant Mehta, H-P Seidel, and Tobias Ritschel. 2017. Deep shading: convolutional neural networks for screen space shading. In Computer graphics forum, Vol. 36. Wiley Online Library, 65–78.
[35]
Ren Ng, Ravi Ramamoorthi, and Pat Hanrahan. 2004. Triple product wavelet integrals for all-frequency relighting. In ACM SIGGRAPH 2004 Papers. 477–487.
[36]
Nvidia. 2023a. NVIDIA Real-Time Denoisers. https://developer.nvidia.com/rtx/ray-tracing/rt-denoisers
[37]
Nvidia. 2023b. OptiX AI-Accelerated Denoiser. https://developer.nvidia.com/optix-denoiser
[38]
Ola Olsson, Markus Billeter, Erik Sintorn, Viktor K?mpe, and Ulf Assarsson. 2015. More efficient virtual shadow maps for many lights. IEEE transactions on visualization and computer graphics 21, 6 (2015), 701–713.
[39]
Roman Prutkin, Anton Kaplanyan, Carsten Dachsbacher, et al. 2012. Reflective Shadow Map Clustering for Real-Time Global Illumination. In Eurographics (Short Papers). 9–12.
[40]
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. 2017. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition. 652–660.
[41]
Nithin Raghavan, Yan Xiao, Kai-En Lin, Tiancheng Sun, Sai Bi, Zexiang Xu, Tzu-Mao Li, and Ravi Ramamoorthi. 2023. Neural Free-Viewpoint Relighting for Glossy Indirect Illumination. In Computer Graphics Forum, Vol. 42. Wiley Online Library, e14885.
[42]
Gilles Rainer, Adrien Bousseau, Tobias Ritschel, and George Drettakis. 2022. Neural precomputed radiance transfer. In Computer Graphics Forum, Vol. 41. Wiley Online Library, 365–378.
[43]
Ravi Ramamoorthi et al. 2009. Precomputation-based rendering. Foundations and Trends? in Computer Graphics and Vision 3, 4 (2009), 281–369.
[44]
Peiran Ren, Jiaping Wang, Minmin Gong, Stephen Lin, Xin Tong, and Baining Guo. 2013. Global illumination with radiance regression functions. ACM Trans. Graph. 32, 4 (2013), 130–1.
[45]
Tobias Ritschel, Thorsten Grosch, Min H Kim, H-P Seidel, Carsten Dachsbacher, and Jan Kautz. 2008. Imperfect shadow maps for efficient computation of indirect illumination. ACM transactions on graphics (tog) 27, 5 (2008), 1–8.
[46]
Tobias Ritschel, Thorsten Grosch, and Hans-Peter Seidel. 2009. Approximating dynamic global illumination in image space. In Proceedings of the 2009 symposium on Interactive 3D graphics and games. 75–82.
[47]
Austin Robison and Peter Shirley. 2009. Image space gathering. In Proceedings of the Conference on High Performance Graphics 2009. 91–98.
[48]
Simon Rodriguez, Thomas Leimk?hler, Siddhant Prakash, Chris Wyman, Peter Shirley, and George Drettakis. 2020. Glossy probe reprojection for interactive global illumination. ACM Transactions on Graphics (TOG) 39, 6 (2020), 1–16.
[49]
Perumaal Shanmugam and Okan Arikan. 2007. Hardware accelerated ambient occlusion techniques on GPUs. In Proceedings of the 2007 symposium on Interactive 3D graphics and games. 73–80.
[50]
Florian Simon, Johannes Hanika, and Carsten Dachsbacher. 2015. Rich-VPLs for improving the versatility of many-light methods. In Computer Graphics Forum, Vol. 34. Wiley Online Library, 575–584.
[51]
Vincent Sitzmann, Michael Zollh?fer, and Gordon Wetzstein. 2019. Scene representation networks: Continuous 3d-structure-aware neural scene representations. Advances in Neural Information Processing Systems 32 (2019).
[52]
Peter-Pike Sloan, Jan Kautz, and John Snyder. 2002. Precomputed radiance transfer for real-time rendering in dynamic, low-frequency lighting environments. In Proceedings of the 29th annual conference on Computer graphics and interactive techniques. 527–536.
[53]
Peter-Pike Sloan, Ben Luna, and John Snyder. 2005. Local, deformable precomputed radiance transfer. ACM Transactions on Graphics (TOG) 24, 3 (2005), 1216–1224.
[54]
Xin Sun, Kun Zhou, Yanyun Chen, Stephen Lin, Jiaoying Shi, and Baining Guo. 2007. Interactive relighting with dynamic BRDFs. In ACM SIGGRAPH 2007 papers. 27–es.
[55]
Ayush Tewari, Justus Thies, Ben Mildenhall, Pratul Srinivasan, Edgar Tretschk, Wang Yifan, Christoph Lassner, Vincent Sitzmann, Ricardo Martin-Brualla, Stephen Lombardi, et al. 2022. Advances in neural rendering. In Computer Graphics Forum, Vol. 41. Wiley Online Library, 703–735.
[56]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ?ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017).
[57]
Delio Vicini, Vladlen Koltun, and Wenzel Jakob. 2019. A learned shape-adaptive subsurface scattering model. ACM Transactions on Graphics (TOG) 38, 4 (2019), 1–15.
[58]
Bruce Walter, Sebastian Fernandez, Adam Arbree, Kavita Bala, Michael Donikian, and Donald P Greenberg. 2005. Lightcuts: a scalable approach to illumination. In ACM SIGGRAPH 2005 Papers. 1098–1107.
[59]
Jiaping Wang, Peiran Ren, Minmin Gong, John Snyder, and Baining Guo. 2009. All-frequency rendering of dynamic, spatially-varying reflectance. In ACM SIGGRAPH Asia 2009 papers. 1–10.
[60]
Lifan Wu, Guangyan Cai, Shuang Zhao, and Ravi Ramamoorthi. 2020. Analytic spherical harmonic gradients for real-time rendering with many polygonal area lights. ACM Transactions on Graphics (TOG) 39, 4 (2020), 134–1.
[61]
Hanggao Xin, Shaokun Zheng, Kun Xu, and Ling-Qi Yan. 2020. Lightweight bilateral convolutional neural networks for interactive single-bounce diffuse indirect illumination. IEEE Transactions on Visualization and Computer Graphics 28, 4 (2020), 1824–1834.
[62]
Bing Xu, Liwen Wu, Milos Hasan, Fujun Luan, Iliyan Georgiev, Zexiang Xu, and Ravi Ramamoorthi. 2023. NeuSample: Importance Sampling for Neural Materials. In ACM SIGGRAPH 2023 Conference Proceedings. 1–10.
[63]
Bing Xu, Junfei Zhang, Rui Wang, Kun Xu, Yong-Liang Yang, Chuan Li, and Rui Tang. 2019. Adversarial Monte Carlo denoising with conditioned auxiliary feature modulation. ACM Trans. Graph. 38, 6 (2019), 224–1.
[64]
Zilin Xu, Zheng Zeng, Lifan Wu, Lu Wang, and Ling-Qi Yan. 2022. Lightweight Neural Basis Functions for All-Frequency Shading. In SIGGRAPH Asia 2022 Conference Papers. 1–9.
[65]
Jiaqi Yu, Yongwei Nie, Chengjiang Long, Wenjun Xu, Qing Zhang, and Guiqing Li. 2021. Monte Carlo denoising via auxiliary feature guided self-attention. ACM Trans. Graph. 40, 6 (2021), 273–1.
[66]
Chong Zeng, Guojun Chen, Yue Dong, Pieter Peers, Hongzhi Wu, and Xin Tong. 2023. Relighting Neural Radiance Fields with Shadow and Highlight Hints. In ACM SIGGRAPH 2023 Conference Proceedings. 1–11.
[67]
Fan Zhang, Hanqiu Sun, Leilei Xu, and Lee Kit Lun. 2006. Parallel-split shadow maps for large-scale virtual environments. In Proceedings of the 2006 ACM international conference on Virtual reality continuum and its applications. 311–318.
[68]
Chuankun Zheng, Yuchi Huo, Shaohua Mo, Zhihua Zhong, Zhizhen Wu, Wei Hua, Rui Wang, and Hujun Bao. 2023. NeLT: Object-oriented Neural Light Transfer. ACM Transactions on Graphics (2023).
[69]
Kun Zhou, Yaohua Hu, Stephen Lin, Baining Guo, and Heung-Yeung Shum. 2005. Precomputed shadow fields for dynamic scenes. In ACM SIGGRAPH 2005 Papers. 1196–1201.
[70]
Junqiu Zhu, Yaoyi Bai, Zilin Xu, Steve Bako, Edgar Vel?zquez-Armend?riz, Lu Wang, Pradeep Sen, Milos Hasan, and Ling-Qi Yan. 2021. Neural complex luminaires: representation and rendering. ACM Trans. Graph. 40, 4 (2021), 57–1.
