“Neural James-Stein Combiner for Unbiased and Biased Renderings” by Gu, Iglesias-Guitian and Moon
Conference:
Type(s):
Title:
- Neural James-Stein Combiner for Unbiased and Biased Renderings
Session/Category Title: Sampling and Reconstruction
Presenter(s)/Author(s):
Abstract:
Unbiased rendering algorithms such as path tracing produce accurate images given a huge number of samples, but in practice, the techniques often leave visually distracting artifacts (i.e., noise) in their rendered images due to a limited time budget. A favored approach for mitigating the noise problem is applying learning-based denoisers to unbiased but noisy rendered images and suppressing the noise while preserving image details. However, such denoising techniques typically introduce a systematic error, i.e., the denoising bias, which does not decline as rapidly when increasing the sample size, unlike the other type of error, i.e., variance. It can technically lead to slow numerical convergence of the denoising techniques. We propose a new combination framework built upon the James-Stein (JS) estimator, which merges a pair of unbiased and biased rendering images, e.g., a path-traced image and its denoised result. Unlike existing post-correction techniques for image denoising, our framework helps an input denoiser have lower errors than its unbiased input without relying on accurate estimation of per-pixel denoising errors. We demonstrate that our framework based on the well-established JS theories allows us to improve the error reduction rates of state-of-the-art learning-based denoisers more robustly than recent post-denoisers.
References:
1. Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2015. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. https://www.tensorflow.org/Software available from tensorflow.org.
2. Jonghee Back, Binh-Son Hua, Toshiya Hachisuka, and Bochang Moon. 2020. Deep combiner for independent and correlated pixel estimates. ACM Trans. Graph. 39, 6 (2020), 12 pages.
3. Steve Bako, Thijs Vogels, Brian McWilliams, Mark Meyer, Jan Novák, Alex Harvill, Pradeep Sen, Tony Derose, and Fabrice Rousselle. 2017. Kernel-predicting convolutional networks for denoising Monte Carlo renderings. ACM Trans. Graph. 36, 4 (2017), 14 pages.
4. Alvin J Baranchik. 1964. Multiple regression and estimation of the mean of a multivariate normal distribution. Technical Report. STANFORD UNIV CALIF.
5. Alvin J Baranchik. 1970. A family of minimax estimators of the mean of a multivariate normal distribution. The Annals of Mathematical Statistics (1970), 642–645.
6. Benedikt Bitterli. 2016. Rendering resources. https://benedikt-bitterli.me/resources/.
7. Benedikt Bitterli, Fabrice Rousselle, Bochang Moon, José A Iglesias-Guitián, David Adler, Kenny Mitchell, Wojciech Jarosz, and Jan Novák. 2016. Nonlinearly weighted first-order regression for denoising monte carlo renderings. In Computer Graphics Forum, Vol. 35. Wiley Online Library, 107–117.
8. Benedikt Bitterli, Chris Wyman, Matt Pharr, Peter Shirley, Aaron Lefohn, and Wojciech Jarosz. 2020. Spatiotemporal Reservoir Resampling for Real-Time Ray Tracing with Dynamic Direct Lighting. ACM Trans. Graph. 39, 4 (2020), 17 pages.
9. Chakravarty R Alla Chaitanya, Anton S Kaplanyan, Christoph Schied, Marco Salvi, Aaron Lefohn, Derek Nowrouzezahrai, and Timo Aila. 2017. Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder. ACM Trans. Graph. 36, 4 (2017), 12 pages.
10. Rudrasis Chakraborty, Yifei Xing, Minxuan Duan, and Stella X Yu. 2020. C-SURE: Shrinkage Estimator and Prototype Classifier for Complex-Valued Deep Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 80–81.
11. Robert L Cook, Thomas Porter, and Loren Carpenter. 1984. Distributed ray tracing. In Proceedings of the 11th annual conference on Computer graphics and interactive techniques. 137–145.
12. Bradley Efron and Carl Morris. 1977. Stein’s paradox in statistics. Scientific American 236, 5 (1977), 119–127.
13. Arthur Firmino, Jeppe Revall Frisvad, and Henrik Wann Jensen. 2022. Progressive Denoising of Monte Carlo Rendered Images. Computer Graphics Forum (2022).
14. Iliyan Georgiev, Jaroslav Krivánek, Tomas Davidovic, and Philipp Slusallek. 2012. Light transport simulation with vertex connection and merging. ACM Trans. Graph. 31, 6 (2012), 10 pages.
15. Michaël Gharbi, Tzu-Mao Li, Miika Aittala, Jaakko Lehtinen, and Frédo Durand. 2019. Sample-Based Monte Carlo Denoising Using a Kernel-Splatting Network. ACM Trans. Graph. 38, 4 (2019), 12 pages.
16. Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative Adversarial Networks. arXiv:1406.2661 [stat.ML]
17. Edwin J Green and William E Strawderman. 1991. A James-Stein type estimator for combining unbiased and possibly biased estimators. J. Amer. Statist. Assoc. 86, 416 (1991), 1001–1006.
18. Toshiya Hachisuka and Henrik Wann Jensen. 2009. Stochastic Progressive Photon Mapping. ACM Trans. Graph. 28, 5 (2009), 8 pages.
19. Jean Hausser and Korbinian Strimmer. 2009. Entropy inference and the James-Stein estimator, with application to nonlinear gene association networks. Journal of Machine Learning Research 10, 7 (2009).
20. Yuchi Huo and Sung-eui Yoon. 2021. A survey on deep learning-based Monte Carlo denoising. Computational Visual Media 7, 2 (2021), 169–185.
21. Wenzel Jakob. 2010. Mitsuba renderer. http://www.mitsuba-renderer.org.
22. James T Kajiya. 1986. The rendering equation. In Proceedings of the 13th annual conference on Computer graphics and interactive techniques. 143–150.
23. Nima Khademi Kalantari, Steve Bako, and Pradeep Sen. 2015. A machine learning approach for filtering Monte Carlo noise. ACM Trans. Graph. 34, 4 (2015), 12 pages.
24. Markus Kettunen, Erik Härkönen, and Jaakko Lehtinen. 2019. Deep Convolutional Reconstruction for Gradient-Domain Rendering. ACM Trans. Graph. 38, 4 (2019), 12 pages.
25. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
26. Eric P. Lafortune and Yves D. Willems. 1993. Bi-directional path tracing. In Proceedings of Third International Conference on Computational Graphics and Visualization Techniques (Compugraphics ’93). Alvor, Portugal, 145–153.
27. Tzu-Mao Li, Yu-Ting Wu, and Yung-Yu Chuang. 2012. SURE-based optimization for adaptive sampling and reconstruction. ACM Trans. Graph. 31, 6 (2012), 9 pages.
28. Zehui Lin, Sheng Li, Xinlu Zeng, Congyi Zhang, Jinzhu Jia, Guoping Wang, and Dinesh Manocha. 2020. CPPM: chi-squared progressive photon mapping. ACM Trans. Graph. 39, 6 (2020), 12 pages.
29. Jonathan H Manton, Vikram Krishnamurthy, and H Vincent Poor. 1998. James-Stein state filtering algorithms. IEEE Transactions on Signal Processing 46, 9 (1998), 2431–2447.
30. Bochang Moon, Nathan Carr, and Sung-Eui Yoon. 2014. Adaptive rendering based on weighted local regression. ACM Trans. Graph. 33, 5 (2014), 14 pages.
31. Bochang Moon, Steven McDonagh, Kenny Mitchell, and Markus Gross. 2016. Adaptive Polynomial Rendering. ACM Trans. Graph. 35, 4 (2016), 10 pages.
32. Jacob Munkberg and Jon Hasselgren. 2020. Neural denoising with layer embeddings. In Computer Graphics Forum, Vol. 39. Wiley Online Library, 1–12.
33. Ryan S Overbeck, Craig Donner, and Ravi Ramamoorthi. 2009. Adaptive wavelet rendering. ACM Trans. Graph. 28, 5 (2009), 12 pages.
34. Matt Pharr. 2018. Guest Editor’s Introduction: Special Issue on Production Rendering. ACM Trans. Graph. 37, 3 (2018), 4 pages.
35. Matt Pharr, Wenzel Jakob, and Greg Humphreys. 2016. Physically based rendering: From theory to implementation. Morgan Kaufmann.
36. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention. Springer, 234–241.
37. Fabrice Rousselle, Claude Knaus, and Matthias Zwicker. 2011. Adaptive sampling and reconstruction using greedy error minimization. ACM Trans. Graph. 30, 6 (2011), 12 pages.
38. Fabrice Rousselle, Claude Knaus, and Matthias Zwicker. 2012. Adaptive Rendering with Non-Local Means Filtering. ACM Trans. Graph. 31, 6 (2012), 11 pages.
39. Fabrice Rousselle, Marco Manzi, and Matthias Zwicker. 2013. Robust denoising using feature and color information. In Computer Graphics Forum, Vol. 32. Wiley Online Library, 121–130.
40. Pradeep Sen and Soheil Darabi. 2012. On filtering the noise from the random parameters in Monte Carlo rendering. ACM Trans. Graph. 31, 3 (2012), 15 pages.
41. Charles Stein and Willard James. 1961. Estimation with quadratic loss. In Proc. 4th Berkeley Symp. Mathematical Statistics Probability, Vol. 1. 361–379.
42. Charles M Stein. 1981. Estimation of the mean of a multivariate normal distribution. The annals of Statistics (1981), 1135–1151.
43. Manu Mathew Thomas, Karthik Vaidyanathan, Gabor Liktor, and Angus G. Forbes. 2020. A Reduced-Precision Network for Image Reconstruction. ACM Trans. Graph. 39, 6 (2020), 12 pages.
44. Thijs Vogels, Fabrice Rousselle, Brian McWilliams, Gerhard Röthlin, Alex Harvill, David Adler, Mark Meyer, and Jan Novák. 2018. Denoising with kernel prediction and asymmetric loss functions. ACM Trans. Graph. 37, 4 (2018), 15 pages.
45. Yue Wu, Brian Tracey, Premkumar Natarajan, and Joseph P. Noonan. 2013. James-Stein Type Center Pixel Weights for Non-Local Means Image Denoising. IEEE Signal Processing Letters 20, 4 (2013), 411–414.
46. Bing Xu, Junfei Zhang, Rui Wang, Kun Xu, Yong-Liang Yang, Chuan Li, and Rui Tang. 2019. Adversarial Monte Carlo denoising with conditioned auxiliary feature modulation. ACM Trans. Graph. 38, 6 (2019), 12 pages.
47. Jiaqi Yu, Yongwei Nie, Chengjiang Long, Wenju Xu, Qing Zhang, and Guiqing Li. 2021. Monte Carlo Denoising via Auxiliary Feature Guided Self-Attention. ACM Trans. Graph. 40, 6 (2021), 13 pages.
48. Shaokun Zheng, Fengshi Zheng, Kun Xu, and Ling-Qi Yan. 2021. Ensemble denoising for Monte Carlo renderings. ACM Trans. Graph. 40, 6 (2021), 17 pages.
49. Matthias Zwicker, Wojciech Jarosz, Jaakko Lehtinen, Bochang Moon, Ravi Ramamoorthi, Fabrice Rousselle, Pradeep Sen, Cyril Soler, and S-E Yoon. 2015. Recent advances in adaptive sampling and reconstruction for Monte Carlo rendering. In Computer graphics forum, Vol. 34. Wiley Online Library, 667–681.
