“Hierarchical neural reconstruction for path guiding using hybrid path and photon samples” by Zhu, Xu, Sun, Kuznetsov, Meyer, et al. …

Path guiding is a promising technique to reduce the variance of path tracing. Although existing online path guiding algorithms can eventually learn good sampling distributions given a large amount of time and samples, the speed of learning becomes a major bottleneck. In this paper, we accelerate the learning of sampling distributions by training a light-weight neural network offline to reconstruct from sparse samples. Uniquely, we design our neural network to directly operate convolutions on a sparse quadtree, which regresses a high-quality hierarchical sampling distribution. Our approach can reconstruct reasonably accurate sampling distributions faster, allowing for efficient path guiding and rendering. In contrast to the recent offline neural path guiding techniques that reconstruct low-resolution 2D images for sampling, our novel hierarchical framework enables more fine-grained directional sampling with less memory usage, effectively advancing the practicality and efficiency of neural path guiding. In addition, we take advantage of hybrid bidirectional samples including both path samples and photons, as we have found this more robust to different light transport scenarios compared to using only one type of sample as in previous work. Experiments on diverse testing scenes demonstrate that our approach often improves rendering results with better visual quality and lower errors. Our framework can also provide the proper balance of speed, memory cost, and robustness.

1. Steve Bako, Mark Meyer, Tony DeRose, and Pradeep Sen. 2019. Offline Deep Importance Sampling for Monte Carlo Path Tracing. In Computer Graphics Forum, Vol. 38. Wiley Online Library, 527–542.Google Scholar
2. Steve Bako, Thijs Vogels, Brian McWilliams, Mark Meyer, Jan Novák, Alex Harvill, Pradeep Sen, Tony Derose, and Fabrice Rousselle. 2017. Kernel-predicting convolutional networks for denoising Monte Carlo renderings. ACM Trans. Graph. 36, 4 (2017), 97–1.Google ScholarDigital Library
3. Benedikt Bitterli. 2016. Rendering resources. https://benedikt-bitterli.me/resources/.Google Scholar
4. LLC Blend Swap. 2016. Blend swap.Google Scholar
5. Chakravarty R Alla Chaitanya, Anton S Kaplanyan, Christoph Schied, Marco Salvi, Aaron Lefohn, Derek Nowrouzezahrai, and Timo Aila. 2017. Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder. ACM Transactions on Graphics (TOG) 36, 4 (2017), 1–12.Google ScholarDigital Library
6. Kashyap Chitta, Jose M Alvarez, and Martial Hebert. 2020. Quadtree Generating Networks: Efficient Hierarchical Scene Parsing with Sparse Convolutions. In The IEEE Winter Conference on Applications of Computer Vision.Google Scholar
7. Stavros Diolatzis, Adrien Gruson, Wenzel Jakob, Derek Nowrouzezahrai, and George Drettakis. 2020. Practical Product Path Guiding Using Linearly Transformed Cosines. In Computer Graphics Forum, Vol. 39. Wiley Online Library, 23–33.Google Scholar
8. TM Evermotion. 2012. Evermotion 3d models.Google Scholar
9. Iliyan Georgiev, Jaroslav Krivánek, Tomas Davidovic, and Philipp Slusallek. 2012. Light transport simulation with vertex connection and merging. ACM Trans. Graph. 31, 6 (2012), 192–1.Google ScholarDigital Library
10. Ben Graham. 2015. Sparse 3D convolutional neural networks. arXiv preprint arXiv:1505.02890 (2015).Google Scholar
11. Benjamin Graham, Martin Engelcke, and Laurens Van Der Maaten. 2018. 3d semantic segmentation with submanifold sparse convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 9224–9232.Google ScholarCross Ref
12. Benjamin Graham and Laurens van der Maaten. 2017. Submanifold sparse convolutional networks. arXiv preprint arXiv:1706.01307 (2017).Google Scholar
13. Jerry Guo, Pablo Bauszat, Jacco Bikker, and Elmar Eisemann. 2018. Primary sample space path guiding. In Eurographics Symposium on Rendering, Vol. 2018. The Eurographics Association, 73–82.Google Scholar
14. Toshiya Hachisuka, Shinji Ogaki, and Henrik Wann Jensen. 2008. Progressive photon mapping. In ACM SIGGRAPH Asia 2008 papers. 1–8.Google Scholar
15. Toshiya Hachisuka, Jacopo Pantaleoni, and Henrik Wann Jensen. 2012. A path space extension for robust light transport simulation. ACM Transactions on Graphics (TOG) 31, 6 (2012), 1–10.Google ScholarDigital Library
16. Sebastian Herholz, Oskar Elek, Jiří Vorba, Hendrik Lensch, and Jaroslav Křivánek. 2016. Product importance sampling for light transport path guiding. In Computer Graphics Forum, Vol. 35. Wiley Online Library, 67–77.Google Scholar
17. Yuchi Huo, Rui Wang, Ruzahng Zheng, Hualin Xu, Hujun Bao, and Sung-Eui Yoon. 2020. Adaptive Incident Radiance Field Sampling and Reconstruction Using Deep Reinforcement Learning. ACM Transactions on Graphics (TOG) 39, 1 (2020), 1–17.Google ScholarDigital Library
18. Wenzel Jakob. 2010. Mitsuba renderer. http://www.mitsuba-renderer.org.Google Scholar
19. Henrik Wann Jensen. 1995. Importance driven path tracing using the photon map. In Eurographics Workshop on Rendering Techniques. Springer, 326–335.Google ScholarCross Ref
20. Henrik Wann Jensen. 1996. Global illumination using photon maps. In Rendering Techniques’ 96. Springer, 21–30.Google Scholar
21. James T Kajiya. 1986. The rendering equation. In Proceedings of the 13th annual conference on Computer graphics and interactive techniques. 143–150.Google ScholarDigital Library
22. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).Google Scholar
23. Claude Knaus and Matthias Zwicker. 2011. Progressive photon mapping: A probabilistic approach. ACM Transactions on Graphics (TOG) 30, 3 (2011), 25.Google ScholarDigital Library
24. Jaroslav Křivánek, Iliyan Georgiev, Toshiya Hachisuka, Petr Vévoda, Martin Šik, Derek Nowrouzezahrai, and Wojciech Jarosz. 2014. Unifying points, beams, and paths in volumetric light transport simulation. ACM Transactions on Graphics (TOG) 33, 4 (2014), 1–13.Google ScholarDigital Library
25. Pradeep Kumar Jayaraman, Jianhan Mei, Jianfei Cai, and Jianmin Zheng. 2018. Quadtree convolutional neural networks. In Proceedings of the European Conference on Computer Vision (ECCV). 546–561.Google ScholarCross Ref
26. Eric P Lafortune and Yves D Willems. 1993. Bi-directional path tracing. (1993).Google Scholar
27. Jun Li, Kai Xu, Siddhartha Chaudhuri, Ersin Yumer, Hao Zhang, and Leonidas Guibas. 2017. Grass: Generative recursive autoencoders for shape structures. ACM Transactions on Graphics (TOG) 36, 4 (2017), 1–14.Google ScholarDigital Library
28. Manyi Li, Akshay Gadi Patil, Kai Xu, Siddhartha Chaudhuri, Owais Khan, Ariel Shamir, Changhe Tu, Baoquan Chen, Daniel Cohen-Or, and Hao Zhang. 2019. Grains: Generative recursive autoencoders for indoor scenes. ACM Transactions on Graphics (TOG) 38, 2 (2019), 1–16.Google ScholarDigital Library
29. Kaichun Mo, Paul Guerrero, Li Yi, Hao Su, Peter Wonka, Niloy J Mitra, and Leonidas J Guibas. 2019. StructureNet: hierarchical graph networks for 3D shape generation. ACM Transactions on Graphics (TOG) 38, 6 (2019), 242.Google ScholarDigital Library
30. Thomas Müller. 2019. “Practical Path Guiding” in Production. In ACM SIGGRAPH Courses: Path Guiding in Production, Chapter 10. ACM, New York, NY, USA, 18:35–18:48. Google ScholarDigital Library
31. Thomas Müller, Markus Gross, and Jan Novák. 2017. Practical path guiding for efficient light-transport simulation. In Computer Graphics Forum, Vol. 36. Wiley Online Library, 91–100.Google Scholar
32. Thomas Müller, Brian McWilliams, Fabrice Rousselle, Markus Gross, and Jan Novák. 2019. Neural importance sampling. ACM Transactions on Graphics (TOG) 38, 5 (2019), 1–19.Google ScholarDigital Library
33. Thomas Müller, Fabrice Rousselle, Alexander Keller, and Jan Novák. 2020. Neural control variates. ACM Transactions on Graphics (TOG) 39, 6 (2020), 1–19.Google ScholarDigital Library
34. Steven G Parker, James Bigler, Andreas Dietrich, Heiko Friedrich, Jared Hoberock, David Luebke, David McAllister, Morgan McGuire, Keith Morley, Austin Robison, et al. 2010. OptiX: a general purpose ray tracing engine. Acm transactions on graphics (tog) 29, 4 (2010), 1–13.Google Scholar
35. Alexander Rath, Pascal Grittmann, Sebastian Herholz, Petr Vévoda, Philipp Slusallek, and Jaroslav Křivánek. 2020. Variance-Aware Path Guiding. ACM Transactions on Graphics (Proceedings of SIGGRAPH 2020) 39, 4 (July 2020), 151:1–151:12. Google ScholarDigital Library
36. Gernot Riegler, Ali Osman Ulusoy, and Andreas Geiger. 2017. Octnet: Learning deep 3d representations at high resolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3577–3586.Google ScholarCross Ref
37. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention. Springer, 234–241.Google ScholarCross Ref
38. Lukas Ruppert, Sebastian Herholz, and Hendrik P. A. Lensch. 2020. Robust Fitting of Parallax-Aware Mixtures for Path Guiding. ACM Transactions on Graphics (TOG) (2020).Google Scholar
39. Peter Shirley, Bretton Wade, Philip M Hubbard, David Zareski, Bruce Walter, and Donald P Greenberg. 1995. Global illumination via density-estimation. In Rendering Techniques’ 95. Springer, 219–230.Google Scholar
40. Turbo Squid. 2020. 3D Models, Plugins, Textures, and more at Turbo Squid.Google Scholar
41. Maxim Tatarchenko, Alexey Dosovitskiy, and Thomas Brox. 2017. Octree generating networks: Efficient convolutional architectures for high-resolution 3d outputs. In Proceedings of the IEEE International Conference on Computer Vision. 2088–2096.Google ScholarCross Ref
42. CG Trader. 2020. Cg trader. URL http://www.cgtrader.com 4 (2020).Google Scholar
43. Eric Veach. 1997. Robust Monte Carlo methods for light transport simulation. Vol. 1610. Stanford University PhD thesis.Google ScholarDigital Library
44. Eric Veach and Leonidas Guibas. 1995a. Bidirectional estimators for light transport. In Photorealistic Rendering Techniques. Springer, 145–167.Google Scholar
45. Eric Veach and Leonidas J Guibas. 1995b. Optimally combining sampling techniques for Monte Carlo rendering. In Proceedings of the 22nd annual conference on Computer graphics and interactive techniques. 419–428.Google ScholarDigital Library
46. Thijs Vogels, Fabrice Rousselle, Brian McWilliams, Gerhard Röthlin, Alex Harvill, David Adler, Mark Meyer, and Jan Novák. 2018. Denoising with kernel prediction and asymmetric loss functions. ACM Transactions on Graphics (TOG) 37, 4 (2018), 1–15.Google ScholarDigital Library
47. Jiří Vorba, Johannes Hanika, Sebastian Herholz, Thomas Müller, Jaroslav Křivánek, and Alexander Keller. 2019. Path Guiding in Production. In ACM SIGGRAPH Courses. ACM, New York, NY, USA, 18:1–18:77. Google ScholarDigital Library
48. Jiří Vorba, Ondřej Karlík, Martin Šik, Tobias Ritschel, and Jaroslav Křivánek. 2014. On-line learning of parametric mixture models for light transport simulation. ACM Transactions on Graphics (TOG) 33, 4 (2014), 1–11.Google ScholarDigital Library
49. Jiří Vorba and Jaroslav Křivánek. 2016. Adjoint-driven Russian roulette and splitting in light transport simulation. ACM Transactions on Graphics (TOG) 35, 4 (2016), 1–11.Google ScholarDigital Library
50. Peng-Shuai Wang, Yang Liu, Yu-Xiao Guo, Chun-Yu Sun, and Xin Tong. 2017. O-CNN: Octree-based Convolutional Neural Networks for 3D Shape Analysis. ACM Transactions on Graphics (SIGGRAPH) 36, 4 (2017).Google ScholarDigital Library
51. Peng-Shuai Wang, Yang Liu, and Xin Tong. 2020. Deep Octree-based CNNs with Output-Guided Skip Connections for 3D Shape and Scene Completion. Computer Vision and Pattern Recognition (CVPR) Workshops.Google ScholarCross Ref
52. Peng-Shuai Wang, Chun-Yu Sun, Yang Liu, and Xin Tong. 2018. Adaptive O-CNN: A Patch-based Deep Representation of 3D Shapes. ACM Transactions on Graphics (SIGGRAPH Asia) 37, 6 (2018).Google Scholar
53. Quan Zheng and Matthias Zwicker. 2019. Learning to importance sample in primary sample space. In Computer Graphics Forum, Vol. 38. Wiley Online Library, 169–179.Google Scholar
54. Shilin Zhu, Zexiang Xu, Henrik Wann Jensen, Hao Su, and Ravi Ramamoorthi. 2020a. Deep Kernel Density Estimation for Photon Mapping. In Computer Graphics Forum, Vol. 39. Wiley-Blackwell.Google Scholar
55. Shilin Zhu, Zexiang Xu, Tiancheng Sun, Alexandr Kuznetsov, Mark Meyer, Henrik Wann Jensen, Hao Su, and Ravi Ramamoorthi. 2020b. Photon-Driven Neural Path Guiding. arXiv preprint arXiv:2010.01775 (2020).Google Scholar