“NeLT: Object-oriented Neural Light Transfer”
Conference:
Type(s):
Title:
- NeLT: Object-oriented Neural Light Transfer
Presenter(s)/Author(s):
Abstract:
Our work presents object-oriented neural light transfer (NeLT), a novel modular neural representation of the dynamic light transportation between an object and the environment. It enables interactive rendering with global illumination for dynamic scenes and achieves comparable quality to the recent advanced real-time denoiser.
References:
[1]
Kara-Ali Aliev, Artem Sevastopolsky, Maria Kolos, Dmitry Ulyanov, and Victor Lempitsky. 2020. Neural point-based graphics. In Computer Vision?ECCV 2020. Lecture Notes in Computer Science, Vol. 12367. Springer, 696?712.
[2]
Steve Bako, Thijs Vogels, Brian McWilliams, Mark Meyer, Jan Nov?k, Alex Harvill, Pradeep Sen, Tony Derose, and Fabrice Rousselle. 2017. Kernel-predicting convolutional networks for denoising Monte Carlo renderings. ACM Transactions on Graphics 36, 4 (2017), Article 97, 1?14.
[3]
Aner Ben-Artzi, Kevin Egan, Fr?do Durand, and Ravi Ramamoorthi. 2008. A precomputed polynomial representation for interactive BRDF editing with global illumination. ACM Transactions on Graphics 27, 2 (2008), 1?13.
[4]
Aner Ben-Artzi, Ryan Overbeck, and Ravi Ramamoorthi. 2006. Real-time BRDF editing in complex lighting. ACM Transactions on Graphics 25, 3 (2006), 945?954.
[5]
Adrian Blumer, Jan Nov?k, Ralf Habel, Derek Nowrouzezahrai, and Wojciech Jarosz. 2016. Reduced aggregate scattering operators for path tracing. Computer Graphics Forum 35 (2016), 461?473.
[6]
Chakravarty R. Alla Chaitanya, Anton S. Kaplanyan, Christoph Schied, Marco Salvi, Aaron Lefohn, Derek Nowrouzezahrai, and Timo Aila. 2017. Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder. ACM Transactions on Graphics 36, 4 (2017), Article 98, 1?12.
[7]
In-Young Cho, Yuchi Huo, and Sung-Eui Yoon. 2021. Weakly-supervised contrastive learning in path manifold for monte carlo image reconstruction. ACM Trans. Graph. 40, 4 (2021), 38?1.
[8]
Carsten Dachsbacher and Marc Stamminger. 2005. Reflective shadow maps. In Proceedings of the 2005 Symposium on Interactive 3D Graphics and Games. 203?231.
[9]
Carsten Dachsbacher, Marc Stamminger, George Drettakis, and Fr?do Durand. 2007. Implicit visibility and antiradiance for interactive global illumination. ACM Transactions on Graphics 26, 3 (2007), 61?es.
[10]
Sayantan Datta, Derek Nowrouzezahrai, Christoph Schied, and Zhao Dong. 2022. Neural shadow mapping. In Proceedings of the ACM SIGGRAPH 2022 Conference (SIGGRAPH?22). 1?9.
[11]
Stavros Diolatzis, Julien Philip, and George Drettakis. 2022. Active exploration for neural global illumination of variable scenes. ACM Transactions on Graphics 41, 5 (2022), Article 171, 18 pages.
[12]
S. M. Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S. Morcos, Marta Garnelo, Avraham Ruderman, et al. 2018. Neural scene representation and rendering. Science 360, 6394 (2018), 1204?1210.
[13]
Hangming Fan, Rui Wang, Yuchi Huo, and Hujun Bao. 2021. Real-time Monte Carlo denoising with weight sharing kernel prediction network. Computer Graphics Forum 40 (2021), 15?27.
[14]
Micha?l Gharbi, Tzu-Mao Li, Miika Aittala, Jaakko Lehtinen, and Fr?do Durand. 2019. Sample-based Monte Carlo denoising using a kernel-splatting network. ACM Transactions on Graphics 38, 4 (2019), 1?12.
[15]
Jonathan Granskog, Fabrice Rousselle, Marios Papas, and Jan Nov?k. 2020. Compositional neural scene representations for shading inference. ACM Transactions on Graphics 39, 4 (2020), Article 135, 13 pages.
[16]
Jonathan Granskog, Till N. Schnabel, Fabrice Rousselle, and Jan Nov?k. 2021. Neural scene graph rendering. ACM Transactions on Graphics 40, 4 (2021), 1?11.
[17]
Jie Guo, Mengtian Li, Quewei Li, Yuting Qiang, Bingyang Hu, Yanwen Guo, and Ling-Qi Yan. 2019. GradNet: Unsupervised deep screened Poisson reconstruction for gradient-domain rendering. ACM Transactions on Graphics 38, 6 (2019), 1?13.
[18]
Michelle Guo, Alireza Fathi, Jiajun Wu, and Thomas Funkhouser. 2020. Object-centric neural scene rendering. arXiv preprint arXiv:2012.08503 (2020).
[19]
David Ha, Andrew M. Dai, and Quoc V. Le. 2017. Hyper networks. In Proceedings of the 5th International Conference on Learning Representations: Conference Track (ICLR?17).
[20]
Saeed Hadadan, Shuhong Chen, and Matthias Zwicker. 2021. Neural radiosity. ACM Transactions on Graphics 40, 6 (2021), 1?11.
[21]
Pedro Hermosilla, Sebastian Maisch, Tobias Ritschel, and Timo Ropinski. 2019. Deep-learning the latent space of light transport. Computer Graphics Forum 38 (2019), 207?217).
[22]
Yuchi Huo, Rui Wang, Ruzahng Zheng, Hualin Xu, Hujun Bao, and Sung-Eui Yoon. 2020. Adaptive incident radiance field sampling and reconstruction using deep reinforcement learning. ACM Transactions on Graphics 39, 1 (2020), 1?17.
[23]
Yuchi Huo and Sung-Eui Yoon. 2021. A survey on deep learning-based Monte Carlo denoising. Computational Visual Media 7 (2021), 169?185.
[24]
Simon Kallweit, Thomas M?ller, Brian McWilliams, Markus Gross, and Jan Nov?k. 2017. Deep scattering: Rendering atmospheric clouds with radiance-predicting neural networks. ACM Transactions on Graphics 36, 6 (2017), 1?11.
[25]
Brian Karis. 2014. High-quality temporal supersampling. In Proceedings of the ACM SIGGRAPH 2014 Conference: Advances in Real-Time Rendering in Games(SIGGRAPH?14).
[26]
Jan Kautz, Jaakko Lehtinen, and Timo Aila. 2004. Hemispherical rasterization for self-shadowing of dynamic objects. In Proceedings of the 15th Eurographics Conference on Rendering Techniques (EGSR?04). 179?184.
[27]
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations (ICLR?15): Conference Track.
[28]
Janne Kontkanen and Samuli Laine. 2005. Ambient occlusion fields. In Proceedings of the 2005 Symposium on Interactive 3D Graphics and Games. 41?48.
[29]
Anders Wang Kristensen, Tomas Akenine-M?ller, and Henrik Wann Jensen. 2005. Precomputed local radiance transfer for real-time lighting design. ACM Transactions on Graphics 24, 3 (2005), 1208?1215.
[30]
Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. 2019. Neural volumes: Learning dynamic renderable volumes from images. ACM Transactions on Graphics 38, 4 (July 2019), Article 65, 14 pages.
[31]
Bradford J. Loos, Lakulish Antani, Kenny Mitchell, Derek Nowrouzezahrai, Wojciech Jarosz, and Peter-Pike Sloan. 2011. Modular radiance transfer. In Proceedings of the 2011 SIGGRAPH Asia Conference. 1?10.
[32]
Bradford J. Loos, Derek Nowrouzezahrai, Wojciech Jarosz, and Peter-Pike Sloan. 2012. Delta radiance transfer. In Proceedings of the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games. 191?196.
[33]
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. 2021. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM 65, 1 (2021), 99?106.
[34]
Thomas M?ller. 2021. Tiny CUDA Neural Network Framework. Retrieved May 18, 2023 from https://github.com/nvlabs/tiny-cuda-nn.
[35]
Thomas M?ller, Fabrice Rousselle, Jan Nov?k, and Alexander Keller. 2021. Real-time neural radiance caching for path tracing. ACM Transactions on Graphics 40, 4 (2021), 1?16.
[36]
Oliver Nalbach, Elena Arabadzhiyska, Dushyant Mehta, H.-P. Seidel, and Tobias Ritschel. 2017. Deep shading: Convolutional neural networks for screen space shading. Computer Graphics Forum 36 (2017), 65?78.
[37]
Michael Niemeyer and Andreas Geiger. 2021. GIRAFFE: Representing scenes as compositional generative neural feature fields. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR?21).
[38]
Julian Ost, Fahim Mannan, Nils Thuerey, Julian Knodt, and Felix Heide. 2021. Neural scene graphs for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2856?2865.
[39]
Steven G. Parker, James Bigler, Andreas Dietrich, Heiko Friedrich, Jared Hoberock, David Luebke, David McAllister, et al. 2010. Optix: A general purpose ray tracing engine. ACM Transactions on Graphics 29, 4 (2010), 1?13.
[40]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, et al. 2019. PyTorch: An imperative style, high-performance deep learning library. In Proceedings of the 33rd International Conference on Neural Information Processing Systems (NIPS?19). 8026?8037.
[41]
Matt Pharr, Wenzel Jakob, and Greg Humphreys. 2020. Physically Based Rendering: From Theory to Implementation (4th ed). MIT Press, Cambridge, MA. https://github.com/mmp/pbrt-v4.
[42]
Konstantinos Rematas and Vittorio Ferrari. 2020. Neural voxel renderer: Learning an accurate and controllable rendering tool. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5417?5427.
[43]
Peiran Ren, Jiaping Wang, Minmin Gong, Stephen Lin, Xin Tong, and Baining Guo. 2013. Global illumination with radiance regression functions. ACM Transactions on Graphics 32, 4 (2013), Article 130, 12 pages.
[44]
Tobias Ritschel, Thorsten Grosch, and Hans-Peter Seidel. 2009. Approximating dynamic global illumination in image space. In Proceedings of the 2009 Symposium on Interactive 3D Graphics and Games. 75?82.
[45]
Paul Sanzenbacher, Lars Mescheder, and Andreas Geiger. 2020. Learning neural light transport. arXiv preprint arXiv:2006.03427 (2020).
[46]
Perumaal Shanmugam and Okan Arikan. 2007. Hardware accelerated ambient occlusion techniques on GPUs. In Proceedings of the 2007 Symposium on Interactive 3D Graphics and Games. 73?80.
[47]
Peter-Pike Sloan, Jan Kautz, and John Snyder. 2002. Precomputed radiance transfer for real-time rendering in dynamic, low-frequency lighting environments. In Proceedings of the 29th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH?02). 527?536.
[48]
Xin Sun, Kun Zhou, Yanyun Chen, Stephen Lin, Jiaoying Shi, and Baining Guo. 2007. Interactive relighting with dynamic BRDFs. ACM Transactions on Graphics 26, 3 (2007), 27?es.
[49]
Maxim Tatarchenko, Alexey Dosovitskiy, and Thomas Brox. 2016. Multi-view 3D models from single images with a convolutional network. In Proceedings of the European Conference on Computer Vision. 322?337.
[50]
Delio Vicini, Vladlen Koltun, and Wenzel Jakob. 2019. A learned shape-adaptive subsurface scattering model. ACM Transactions on Graphics 38, 4 (2019), 1?15.
[51]
Thijs Vogels, Fabrice Rousselle, Brian McWilliams, Gerhard R?thlin, Alex Harvill, David Adler, Mark Meyer, and Jan Nov?k. 2018. Denoising with kernelprediction and asymmetric loss functions. ACM Transactions on Graphics 37, 4 (2018), 1?15.
[52]
Bruce Walter, Sebastian Fernandez, Adam Arbree, Kavita Bala, Michael Donikian, and Donald P. Greenberg. 2005. Lightcuts: A scalable approach to illumination. ACM Transactions on Graphics 24, 3 (2005), 1098?1107.
[53]
Bruce Walter, Stephen R. Marschner, Hongsong Li, and Kenneth E. Torrance. 2007. Microfacet models for refraction through rough surfaces. In Proceedings of the 18th Eurographics Conference on Rendering Techniques (EGSR?07). 195?206.
[54]
Hanggao Xin, Shaokun Zheng, Kun Xu, and Ling-Qi Yan. 2022. Lightweight bilateral convolutional neural networks for interactive single-bounce diffuse indirect illumination. IEEE Transactions on Visualization and Computer Graphics 28, 4 (2022), 1824?1834.
[55]
Bing Xu, Junfei Zhang, Rui Wang, Kun Xu, Yong-Liang Yang, Chuan Li, and Rui Tang. 2019. Adversarial Monte Carlo denoising with conditioned auxiliary feature modulation. ACM Transactions on Graphics 38, 6 (2019), Article 224, 12 pages.
[56]
Jiaqi Yu, Yongwei Nie, Chengjiang Long, Wenjun Xu, Qing Zhang, and Guiqing Li. 2021. Monte Carlo denoising via auxiliary feature guided self-attention. ACM Transactions on Graphics 40, 6 (2021), Article 273, 13 pages.
[57]
Chuankun Zheng, Ruzhang Zheng, Rui Wang, Shuang Zhao, and Hujun Bao. 2021. A compact representation of measured BRDFs using neural processes. ACM Transactions on Graphics 41, 2 (2021), 1?15.
[58]
Kun Zhou, Yaohua Hu, Stephen Lin, Baining Guo, and Heung-Yeung Shum. 2005. Precomputed shadow fields for dynamic scenes. ACM Transactions on Graphics 24, 3 (2005), 1196?1201.
[59]
Junqiu Zhu, Yaoyi Bai, Zilin Xu, Steve Bako, Edgar Vel?zquez-Armend?riz, Lu Wang, Pradeep Sen, Milo? Ha?an, and Ling-Qi Yan. 2021. Neural complex luminaires: Representation and rendering. ACM Transactions on Graphics 40, 4 (2021), 1?12.
