“Neural Partitioning Pyramids for Denoising Monte Carlo Renderings” by Wolski, Myszkowski, Seidel and Mantiuk
Conference:
Type(s):
Title:
- Neural Partitioning Pyramids for Denoising Monte Carlo Renderings
Session/Category Title: Real-time Rendering: Gotta Go Fast!
Presenter(s)/Author(s):
Moderator(s):
Abstract:
Recent advancements in hardware-accelerated raytracing made it possible to achieve interactive framerates even for algorithms previously considered offline, such as path tracing. Interactive path tracing pipelines rely heavily on spatiotemporal denoising to produce a high-quality output from low-sample-count renderings. Such denoising is typically implemented as multiscale-kernel-based filters driven by lightweight U-Nets operating on pixels, and encoders operating on samples. In this work, we present a novel kernel architecture in the line of low-pass pyramid filters. Our architecture avoids the issues with the low-frequency response of previous such filters, resolving ringing, blotchiness, and box-shaped artefacts while improving overall detail. Instead of using classical downsampling and upsampling approaches, which are prone to aliasing, we let our weight predictor networks learn to partition the input radiance between pyramidal layers, predict kernels for denoising each partitioned and downscaled image, and then guide the upsampling process when combining layers. We present failure cases of pyramidal scale-composition in previous work and, through Fourier analysis, show how our method resolves them. Finally, we demonstrate state-of-the-art denoising performance.
References:
1. Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mane, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viegas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems. (2016).
2. Pontus Andersson, Jim Nilsson, Tomas Akenine-Möller, Magnus Oskarsson, Kalle Åström, and Mark D. Fairchild. 2020. FLIP: A Difference Evaluator for Alternating Images. Proc. ACM Comput. Graph. Interact. Tech. 3, 2, Article 15 (aug 2020), 23 pages. https://doi.org/10.1145/3406183
3. Jonghee Back, Binh-Son Hua, Toshiya Hachisuka, and Bochang Moon. 2022. Self-Supervised Post-Correction for Monte Carlo Denoising. In ACM SIGGRAPH 2022 Conference Proceedings (Vancouver, BC, Canada) (SIGGRAPH ’22). Association for Computing Machinery, New York, NY, USA, Article 18, 8 pages. https://doi.org/10.1145/3528233.3530730
4. Steve Bako, Thijs Vogels, Brian Mcwilliams, Mark Meyer, Jan NováK, Alex Harvill, Pradeep Sen, Tony Derose, and Fabrice Rousselle. 2017. Kernel-Predicting Convolutional Networks for Denoising Monte Carlo Renderings. ACM Trans. Graph. 36, 4, Article 97 (jul 2017), 14 pages. https://doi.org/10.1145/3072959.3073708
5. Amin Banitalebi-Dehkordi, Maryam Azimi, Mahsa T. Pourazad, and Panos Nasiopoulos. 2016. Visual Saliency Aided High Dynamic Range (HDR) Video Quality Metrics. In 2016 IEEE International Conference on Communications Workshops (ICC). 486–491. https://doi.org/10.1109/iccw.2016.7503834
6. Benedikt Bitterli. 2016. Rendering resources. https://benedikt-bitterli.me/resources/.
7. Peter J Burt and Edward H Adelson. 1987. The Laplacian Pyramid as a Compact Image Code. In Readings in computer vision. Elsevier, 671–679. https://doi.org/10.1016/b978-0-08-051581-6.50065-9
8. Chakravarty R. Alla Chaitanya, Anton S. Kaplanyan, Christoph Schied, Marco Salvi, Aaron Lefohn, Derek Nowrouzezahrai, and Timo Aila. 2017. Interactive Reconstruction of Monte Carlo Image Sequences Using a Recurrent Denoising Autoencoder. ACM Trans. Graph. 36, 4, Article 98 (jul 2017), 12 pages. https://doi.org/10.1145/3072959.3073601
9. In-Young Cho, Yuchi Huo, and Sung-Eui Yoon. 2021. Weakly-Supervised Contrastive Learning in Path Manifold for Monte Carlo Image Reconstruction. ACM Trans. Graph. 40, 4, Article 38 (jul 2021), 14 pages. https://doi.org/10.1145/3450626.3459876
10. Mauricio Delbracio, Pablo Musé, Antoni Buades, Julien Chauvier, Nicholas Phelps, and Jean-Michel Morel. 2014. Boosting Monte Carlo Rendering by Ray Histogram Fusion. ACM Trans. Graph. 33, 1, Article 8 (feb 2014), 15 pages. https://doi.org/10.1145/2532708
11. Hangming Fan, Rui Wang, Yuchi Huo, and Hujun Bao. 2021. Real-Time Monte Carlo Denoising with Weight Sharing Kernel Prediction Network. Computer Graphics Forum 40, 4 (2021), 15–27. https://doi.org/10.1111/cgf.14338
12. Siyuan Fu, Yifan Lu, Xiao Hua Zhang, and Ning Xie. 2021. Monte Carlo Denoising with a Sparse Auxiliary Feature Encoder. In SIGGRAPH Asia 2021 Posters (Tokyo, Japan) (SA ’21 Posters). Association for Computing Machinery, New York, NY, USA, Article 10, 2 pages. https://doi.org/10.1145/3476124.3488631
13. Michaël Gharbi, Tzu-Mao Li, Miika Aittala, Jaakko Lehtinen, and Frédo Durand. 2019. Sample-Based Monte Carlo Denoising Using a Kernel-Splatting Network. ACM Trans. Graph. 38, 4, Article 125 (jul 2019), 12 pages. https://doi.org/10.1145/3306346.3322954
14. Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT press.
15. Alex Graves. 2012. Long Short-Term Memory. Springer Berlin Heidelberg, Berlin, Heidelberg, 37–45. https://doi.org/10.1007/978-3-642-24797-2_4
16. J. Hasselgren, J. Munkberg, M. Salvi, A. Patney, and A. Lefohn. 2020. Neural Temporal Adaptive Sampling and Denoising. Computer Graphics Forum 39, 2 (2020), 147–155. https://doi.org/10.1111/cgf.13919
17. Stephen Hill. 2022. ACES Tone Mapping Operator. https://github.com/TheRealMJP/BakingLab/blob/master/BakingLab/ACES.hlsl.
18. Yuchi Huo and Sung-eui Yoon. 2021. A Survey on Deep Learning-Based Monte Carlo Denoising. Computational Visual Media 7, 2 (01 Jun 2021), 169–185. https://doi.org/10.1007/s41095-021-0209-9
19. Mustafa Işık, Krishna Mullia, Matthew Fisher, Jonathan Eisenmann, and Michaël Gharbi. 2021. Interactive Monte Carlo Denoising Using Affinity of Neural Features. ACM Trans. Graph. 40, 4, Article 37 (jul 2021), 13 pages. https://doi.org/10.1145/3450626.3459793
20. Intel. 2022. Intel Open Image Denoise. https://www.openimagedenoise.org/.
21. Simon Kallweit, Petrik Clarberg, Craig Kolb, Tom’aš Davidovič, Kai-Hwa Yao, Theresa Foley, Yong He, Lifan Wu, Lucy Chen, Tomas Akenine-Möller, Chris Wyman, Cyril Crassin, and Nir Benty. 2022. The Falcor Rendering Framework. https://github.com/NVIDIAGameWorks/Falcor https://github.com/NVIDIAGameWorks/Falcor.
22. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. (2014).
23. Matias Koskela, Kalle Immonen, Markku Mäkitalo, Alessandro Foi, Timo Viitanen, Pekka Jääskeläinen, Heikki Kultala, and Jarmo Takala. 2019. Blockwise Multi-Order Feature Regression for Real-Time Path-Tracing Reconstruction. ACM Trans. Graph. 38, 5, Article 138 (jun 2019), 14 pages. https://doi.org/10.1145/3269978
24. P Kozlowski and T Cheblokov. 2021. ReLAX: A Denoiser Tailored to Work with the ReSTIR Algorithm. https://www.nvidia.com/en-us/on-demand/session/gtcspring21-s32759/
25. Weiheng Lin, Beibei Wang, Jian Yang, Lu Wang, and Ling-Qi Yan. 2021. Path-based Monte Carlo Denoising Using a Three-Scale Neural Network. Computer Graphics Forum 40, 1 (2021), 369–381. https://doi.org/10.1111/cgf.14194
26. Hongying Liu, Zhubo Ruan, Peng Zhao, Chao Dong, Fanhua Shang, Yuanyuan Liu, Linlin Yang, and Radu Timofte. 2022b. Video Super-Resolution Based on Deep Learning: A Comprehensive Survey. Artificial Intelligence Review 55, 8 (01 Dec 2022), 5981–6035. https://doi.org/10.1007/s10462-022-10147-y
27. Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. 2022a. A ConvNet for the 2020s. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 11966–11976. https://doi.org/10.1109/cvpr52688.2022.01167
28. Yifan Lu, Siyuan Fu, Xiao Hua Zhang, and Ning Xie. 2021. Denoising Monte Carlo Renderings via a Multi-Scale Featured Dual-Residual GAN. The Visual Computer 37, 9 (01 Sep 2021), 2513–2525. https://doi.org/10.1007/s00371-021-02204-4
29. YiFan Lu, Ning Xie, and Heng Tao Shen. 2020. DMCR-GAN: Adversarial Denoising for Monte Carlo Renderings with Residual Attention Networks and Hierarchical Features Modulation of Auxiliary Buffers. In SIGGRAPH Asia 2020 Technical Communications (Virtual Event, Republic of Korea) (SA ’20). Association for Computing Machinery, New York, NY, USA, Article 5, 4 pages. https://doi.org/10.1145/3410700.3425426
30. Rafał K. Mantiuk, Gyorgy Denes, Alexandre Chapiro, Anton Kaplanyan, Gizem Rufo, Romain Bachy, Trisha Lian, and Anjul Patney. 2021. FovVideoVDP: A Visible Difference Predictor for Wide Field-of-View Video. ACM Trans. Graph. 40, 4, Article 49 (jul 2021), 19 pages. https://doi.org/10.1145/3450626.3459831
31. Xiaoxu Meng, Quan Zheng, Amitabh Varshney, Gurprit Singh, and Matthias Zwicker. 2020. Real-time Monte Carlo Denoising with the Neural Bilateral Grid. In Eurographics Symposium on Rendering – DL-only Track, Carsten Dachsbacher and Matt Pharr (Eds.). The Eurographics Association. https://doi.org/10.2312/sr.20201133
32. Jacob Munkberg and Jon Hasselgren. 2020. Neural Denoising with Layer Embeddings. Computer Graphics Forum 39, 4 (2020), 1–12. https://doi.org/10.1111/cgf.14049
33. Nvidia. 2017. Amazon Lumberyard Bistro, Open Research Content Archive (ORCA). http://developer.nvidia.com/orca/amazon-lumberyard-bistro
34. Nvidia. 2022a. NVIDIA Real-Time Denoisers. https://developer.nvidia.com/rtx/ray-tracing/rt-denoisers.
35. Nvidia. 2022b. OptiX AI-Accelerated Denoiser. https://developer.nvidia.com/optix-denoiser.
36. Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M. Susskind. 2021. Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV). 10892–10902. https://doi.org/10.1109/iccv48922.2021.01073
37. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-Net: Convolutional Networks for Biomedical Image Segmentation. In Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, Nassir Navab, Joachim Hornegger, William M. Wells, and Alejandro F. Frangi (Eds.). Springer International Publishing, Cham, 234–241. https://doi.org/10.1007/978-3-319-24574-4_28
38. Amit Sabne. 2020. XLA : Compiling Machine Learning for Peak Performance.
39. Christoph Schied, Anton Kaplanyan, Chris Wyman, Anjul Patney, Chakravarty R. Alla Chaitanya, John Burgess, Shiqiu Liu, Carsten Dachsbacher, Aaron Lefohn, and Marco Salvi. 2017. Spatiotemporal Variance-Guided Filtering: Real-Time Reconstruction for Path-Traced Global Illumination. In Proceedings of High Performance Graphics (Los Angeles, California) (HPG ’17). Association for Computing Machinery, New York, NY, USA, Article 2, 12 pages. https://doi.org/10.1145/3105762.3105770
40. Richard S Sutton and Andrew G Barto. 2018. Reinforcement Learning: An Introduction. MIT press.
41. Manu Mathew Thomas, Gabor Liktor, Christoph Peters, Sungye Kim, Karthik Vaidyanathan, and Angus G. Forbes. 2022. Temporally Stable Real-Time Joint Neural Denoising and Supersampling. Proc. ACM Comput. Graph. Interact. Tech. 5, 3, Article 21 (jul 2022), 22 pages. https://doi.org/10.1145/3543870
42. C. Tomasi and R. Manduchi. 1998. Bilateral Filtering for Gray and Color Images. In Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271). 839–846. https://doi.org/10.1109/iccv.1998.710815
43. Thijs Vogels, Fabrice Rousselle, Brian Mcwilliams, Gerhard Röthlin, Alex Harvill, David Adler, Mark Meyer, and Jan Novák. 2018. Denoising with Kernel Prediction and Asymmetric Loss Functions. ACM Trans. Graph. 37, 4, Article 124 (jul 2018), 15 pages. https://doi.org/10.1145/3197517.3201388
44. Z. Wang, E.P. Simoncelli, and A.C. Bovik. 2003. Multiscale Structural Similarity for Image Quality Assessment. In The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003, Vol. 2. 1398–1402 Vol.2. https://doi.org/10.1109/ACSSC.2003.1292216
45. Xinyue Wei, Haozhi Huang, Yujin Shi, Hongliang Yuan, Li Shen, and Jue Wang. 2021. End-to-End Adaptive Monte Carlo Denoising and Super-Resolution. arXiv (2021).
46. Mike Winkelmann. 2019. Zero-Day, Open Research Content Archive (ORCA). https://developer.nvidia.com/orca/beeple-zero-day
47. Bing Xu, Junfei Zhang, Rui Wang, Kun Xu, Yong-Liang Yang, Chuan Li, and Rui Tang. 2019. Adversarial Monte Carlo Denoising with Conditioned Auxiliary Feature Modulation. ACM Trans. Graph. 38, 6, Article 224 (nov 2019), 12 pages. https://doi.org/10.1145/3355089.3356547
48. Jiaqi Yu, Yongwei Nie, Chengjiang Long, Wenju Xu, Qing Zhang, and Guiqing Li. 2021. Monte Carlo Denoising via Auxiliary Feature Guided Self-Attention. ACM Trans. Graph. 40, 6, Article 273 (dec 2021), 13 pages. https://doi.org/10.1145/3478513.3480565
49. Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming–Hsuan Yang. 2022. Restormer: Efficient Transformer for High-Resolution Image Restoration. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 5718–5729. https://doi.org/10.1109/cvpr52688.2022.00564
50. Xianyao Zhang, Marco Manzi, Thijs Vogels, Henrik Dahlberg, Markus Gross, and Marios Papas. 2021. Deep Compositional Denoising for High-quality Monte Carlo Rendering. Computer Graphics Forum 40, 4 (2021), 1–13. https://doi.org/10.1111/cgf.14337
51. Dmitry Zhdan. 2021. ReBLUR: A Hierarchical Recurrent Denoiser. Apress, Berkeley, CA, 823–844. https://doi.org/10.1007/978-1-4842-7185-8_49
52. Shaokun Zheng, Fengshi Zheng, Kun Xu, and Ling-Qi Yan. 2021. Ensemble Denoising for Monte Carlo Renderings. ACM Trans. Graph. 40, 6, Article 274 (dec 2021), 17 pages. https://doi.org/10.1145/3478513.3480510
53. M. Zwicker, W. Jarosz, J. Lehtinen, B. Moon, R. Ramamoorthi, F. Rousselle, P. Sen, C. Soler, and S.-E. Yoon. 2015. Recent Advances in Adaptive Sampling and Reconstruction for Monte Carlo Rendering. Computer Graphics Forum 34, 2 (2015), 667–681. https://doi.org/10.1111/cgf.12592
