“Temporally Stable Metropolis Light Transport Denoising Using Recurrent Transformer Blocks”
Conference:
Type(s):
Title:
- Temporally Stable Metropolis Light Transport Denoising Using Recurrent Transformer Blocks
Presenter(s)/Author(s):
Abstract:
We propose a learning-based denoising method for Metropolis Light Transport (MLT) based on recurrent Transformer blocks. We show that our Transformer architecture can more effectively resolve the correlation artifacts compared to the blending-based approaches used in previous work.
References:
[1]
Michael Ashikhmin, Simon Premo?e, Peter Shirley, and Brian Smits. 2001. A variance analysis of the Metropolis light transport algorithm. Computers & Graphics 25, 2 (2001), 287–294.
[2]
Jonghee Back, Binh-Son Hua, Toshiya Hachisuka, and Bochang Moon. 2020. Deep combiner for independent and correlated pixel estimates. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 39, 6 (2020), 242–1.
[3]
Jonghee Back, Binh-Son Hua, Toshiya Hachisuka, and Bochang Moon. 2023. Input-Dependent Uncorrelated Weighting for Monte Carlo Denoising. In SIGGRAPH Asia Conference Proceedings. 1–10.
[4]
Steve Bako, Thijs Vogels, Brian McWilliams, Mark Meyer, Jan Nov?k, Alex Harvill, Pradeep Sen, Tony Derose, and Fabrice Rousselle. 2017. Kernel-predicting convolutional networks for denoising Monte Carlo renderings. ACM Trans. Graph. (Proc. SIGGRAPH) 36, 4 (2017), 97:1–97:14.
[5]
Martin Balint, Krzysztof Wolski, Karol Myszkowski, Hans-Peter Seidel, and Rafa? Mantiuk. 2023. Neural partitioning pyramids for denoising Monte Carlo renderings. 1–11.
[6]
Thomas Bashford-Rogers, Lu?s Paulo Santos, Demetris Marnerides, and Kurt Debattista. 2021. Ensemble Metropolis Light Transport. ACM Trans. Graph. 41, 1, Article 5 (2021), 15 pages.
[7]
Benedikt Bitterli. 2016. VeachAjar. https://benedikt-bitterli.me/resources/.
[8]
Benedikt Bitterli and Wojciech Jarosz. 2019. Selectively Metropolised Monte Carlo Light Transport Simulation. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 38, 6, Article 153 (2019), 10 pages.
[9]
Chakravarty R. Alla Chaitanya, Anton S. Kaplanyan, Christoph Schied, Marco Salvi, Aaron Lefohn, Derek Nowrouzezahrai, and Timo Aila. 2017. Interactive Reconstruction of Monte Carlo Image Sequences Using a Recurrent Denoising Autoencoder. ACM Trans. Graph. (Proc. SIGGRAPH) 36, 4, Article 98 (2017), 12 pages.
[10]
David Cline, Justin Talbot, and Parris Egbert. 2005. Energy Redistribution Path Tracing. ACM Trans. Graph. (Proc. SIGGRAPH) 24, 3 (2005), 1186–1195.
[11]
Markus Ebke. 2021. LightSheet. https://benedikt-bitterli.me/resources/.
[12]
Hangming Fan, Rui Wang, Yuchi Huo, and Hujun Bao. 2021. Real-time Monte Carlo denoising with weight sharing kernel prediction network. Comput. Graph. Forum 40, 4 (2021), 15–27.
[13]
Micha?l Gharbi, Tzu-Mao Li, Miika Aittala, Jaakko Lehtinen, and Fr?do Durand. 2019. Sample-based Monte Carlo Denoising Using a Kernel-splatting Network. ACM Trans. Graph. (Proc. SIGGRAPH) 38, 4, Article 125 (2019), 12 pages.
[14]
Adrien Gruson, Rex West, and Toshiya Hachisuka. 2020. Stratified Markov Chain Monte Carlo Light Transport. Comput. Graph. Forum (Proc. Eurographics) 39, 2 (2020), 351–362.
[15]
Toshiya Hachisuka, Anton S Kaplanyan, and Carsten Dachsbacher. 2014. Multiplexed Metropolis light transport. ACM Trans. Graph. (Proc. SIGGRAPH) 33, 4, Article 100 (2014), 10 pages.
[16]
Jon Hasselgren, Jacob Munkberg, Marco Salvi, Anjul Patney, and Aaron Lefohn. 2020. Neural temporal adaptive sampling and denoising. Comput. Graph. Forum (Proc. Eurographics) 39, 2 (2020), 147–155.
[17]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Computer Vision and Pattern Recognition. 770–778.
[18]
Mustafa I??k, Krishna Mullia, Matthew Fisher, Jonathan Eisenmann, and Micha?l Gharbi. 2021. Interactive Monte Carlo denoising using affinity of neural features. ACM Trans. Graph. (Proc. SIGGRAPH) 40, 4, Article 37 (2021), 13 pages.
[19]
Wenzel Jakob. 2010. Mitsuba renderer. http://www.mitsuba-renderer.org.
[20]
Wenzel Jakob and Steve Marschner. 2012. Manifold exploration: a Markov Chain Monte Carlo technique for rendering scenes with difficult specular transport. ACM Trans. Graph. (Proc. SIGGRAPH) 31, 4, Article 58 (2012), 13 pages.
[21]
Nima Khademi Kalantari, Steve Bako, and Pradeep Sen. 2015. A Machine Learning Approach for Filtering Monte Carlo Noise. ACM Trans. Graph. (Proc. SIGGRAPH) 34, 4, Article 122 (2015), 12 pages.
[22]
Csaba Kelemen, L?szl? Szirmay-Kalos, Gy?rgy Antal, and Ferenc Csonka. 2002. A simple and robust mutation strategy for the Metropolis light transport algorithm. Comput. Graph. Forum (Proc. Eurographics) 21, 3 (2002), 531–540.
[23]
Diederick P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. (2015).
[24]
Shinya Kitaoka, Yoshifumi Kitamura, and Fumio Kishino. 2009. Replica Exchange Light Transport. Comput. Graph. Forum 28, 8 (2009), 2330–2342.
[25]
Matias Koskela, Kalle Immonen, Markku M?kitalo, Alessandro Foi, Timo Viitanen, Pekka J??skel?inen, Heikki Kultala, and Jarmo Takala. 2019. Blockwise multi-order feature regression for real-time path-tracing reconstruction. ACM Trans. Graph. 38, 5 (2019), 1–14.
[26]
Yu-Chi Lai, Feng Liu, and Charles Dyer. 2009. Physically-based Animation Rendering with Markov Chain Monte Carlo. Technical Report UW-CS-TR-1653. University of Wisconsin – Madison Computer Sciences Department.
[27]
Zhengqin Li, Ting-Wei Yu, Shen Sang, Sarah Wang, Meng Song, Yuhan Liu, Yu-Ying Yeh, Rui Zhu, Nitesh Gundavarapu, Jia Shi, et al. 2021. OpenRooms: An open framework for photorealistic indoor scene datasets. In Computer Vision and Pattern Recognition. 7190–7199.
[28]
Jingyun Liang, Jiezhang Cao, Yuchen Fan, Kai Zhang, Rakesh Ranjan, Yawei Li, Radu Timofte, and Luc Van Gool. 2022a. VRT: A video restoration transformer. arXiv preprint arXiv:2201.12288 (2022).
[29]
Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. 2021. SwinIR: Image restoration using swin transformer. In International Conference on Computer Vision. 1833–1844.
[30]
Jingyun Liang, Yuchen Fan, Xiaoyu Xiang, Rakesh Ranjan, Eddy Ilg, Simon Green, Jiezhang Cao, Kai Zhang, Radu Timofte, and Luc V Gool. 2022b. Recurrent video restoration transformer with guided deformable attention. Advances in Neural Information Processing Systems 35 (2022), 378–393.
[31]
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021. Swin transformer: Hierarchical vision transformer using shifted windows. In International Conference on Computer Vision. 10012–10022.
[32]
Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. 2022. A ConvNet for the 2020s. In Computer Vision and Pattern Recognition. 11976–11986.
[33]
Fujun Luan, Shuang Zhao, Kavita Bala, and Ioannis Gkioulekas. 2020. Langevin Monte Carlo Rendering with Gradient-Based Adaptation. ACM Trans. Graph. (Proc. SIGGRAPH) 39, 4, Article 140 (2020), 16 pages.
[34]
Rafa? K Mantiuk, Gyorgy Denes, Alexandre Chapiro, Anton Kaplanyan, Gizem Rufo, Romain Bachy, Trisha Lian, and Anjul Patney. 2021. FovVideoVDP: A visible difference predictor for wide field-of-view video. ACM Trans. Graph. (Proc. SIGGRAPH) 40, 4, Article 49 (2021), 19 pages.
[35]
Soham Uday Mehta, Brandon Wang, and Ravi Ramamoorthi. 2012. Axis-aligned filtering for interactive sampled soft shadows. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 31, 6 (2012), 1–10.
[36]
Xiaoxu Meng, Quan Zheng, Amitabh Varshney, Gurprit Singh, and Matthias Zwicker. 2020. Real-time Monte Carlo Denoising with the Neural Bilateral Grid. In Eurographics Symposium on Rendering – DL-only Track. 13–24.
[37]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems. 8024–8035.
[38]
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-assisted Intervention. 234–241.
[39]
Christoph Schied, Anton Kaplanyan, Chris Wyman, Anjul Patney, Chakravarty R Alla Chaitanya, John Burgess, Shiqiu Liu, Carsten Dachsbacher, Aaron Lefohn, and Marco Salvi. 2017. Spatiotemporal variance-guided filtering: real-time reconstruction for path-traced global illumination. In High Performance Graphics. 1–12.
[40]
Manu Mathew Thomas, Gabor Liktor, Christoph Peters, Sungye Kim, Karthik Vaidyanathan, and Angus G Forbes. 2022. Temporally stable real-time joint neural denoising and supersampling. Proceedings of the ACM on Computer Graphics and Interactive Techniques (Proc. HPG) 5, 3 (2022), 1–22.
[41]
Hugo Touvron, Matthiegu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herv? J?gou. 2021. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning. 10347–10357.
[42]
Joran Van de Woestijne, Roald Frederickx, Niels Billen, and Philip Dutr?. 2017. Temporal coherence for Metropolis light transport. In Eurographics Symposium on Rendering – Experimental Ideas & Implementations. 55–63.
[43]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ?ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017).
[44]
Eric Veach. 1998. Robust Monte Carlo methods for light transport simulation. Stanford University.
[45]
Eric Veach and Leonidas J Guibas. 1997. Metropolis light transport. In SIGGRAPH. 65–76.
[46]
Thijs Vogels, Fabrice Rousselle, Brian McWilliams, Gerhard R?thlin, Alex Harvill, David Adler, Mark Meyer, and Jan Nov?k. 2018. Denoising with kernel prediction and asymmetric loss functions. ACM Trans. Graph. (Proc. SIGGRAPH) 37, 4 (2018), 1–15.
[47]
Zhendong Wang, Xiaodong Cun, Jianmin Bao, Wengang Zhou, Jianzhuang Liu, and Houqiang Li. 2022. Uformer: A general U-shaped transformer for image restoration. In Computer Vision and Pattern Recognition. 17683–17693.
[48]
Lifan Wu, Ling-Qi Yan, Alexandr Kuznetsov, and Ravi Ramamoorthi. 2017. Multiple axis-aligned filters for rendering of combined distribution effects. Comput. Graph. Forum (Proc. EGSR) 36, 4 (2017), 155–166.
[49]
Lei Xiao, Salah Nouri, Matt Chapman, Alexander Fix, Douglas Lanman, and Anton Kaplanyan. 2020. Neural Supersampling for Real-time Rendering. ACM Trans. Graph. (Proc. SIGGRAPH) 39, 4 (2020).
[50]
Ling-Qi Yan, Soham Uday Mehta, Ravi Ramamoorthi, and Fredo Durand. 2015. Fast 4D sheared filtering for interactive rendering of distribution effects. ACM Trans. Graph. 35, 1 (2015), 1–13.
[51]
Jiaqi Yu, Yongwei Nie, Chengjiang Long, Wenjun Xu, Qing Zhang, and Guiqing Li. 2021. Monte Carlo denoising via auxiliary feature guided self-attention. ACM Trans. Graph. 40, 6 (2021), 273–1.
[52]
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. 2022. Restormer: Efficient transformer for high-resolution image restoration. In Computer Vision and Pattern Recognition. 5728–5739.
[53]
Henning Zimmer, Fabrice Rousselle, Wenzel Jakob, Oliver Wang, David Adler, Wojciech Jarosz, Olga Sorkine-Hornung, and Alexander Sorkine-Hornung. 2015. Path-space motion estimation and decomposition for robust animation filtering. Comput. Graph. Forum (Proc. EGSR) 34, 4 (2015), 131–142.
[54]
Tobias Zirr and Carsten Dachsbacher. 2020. Path differential-informed stratified MCMC and adaptive forward path sampling. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 39, 6 (2020), 1–19.
[55]
Matthias Zwicker, Wojciech Jarosz, Jaakko Lehtinen, Bochang Moon, Ravi Ramamoorthi, Fabrice Rousselle, Pradeep Sen, Cyril Soler, and S-E Yoon. 2015. Recent advances in adaptive sampling and reconstruction for Monte Carlo rendering. Comput. Graph. Forum (Proc. Eurographics STAR) 34, 2 (2015), 667–681.
