“Kernel-predicting convolutional networks for denoising Monte Carlo renderings”

  • ©Steve Bako, Thijs Vogels, Brian Mcwilliams, Mark Meyer, Jan Novák, Alex Harvill, Pradeep Sen, Tony DeRose, and Fabrice Rousselle

Conference:


Type:


Title:

    Kernel-predicting convolutional networks for denoising Monte Carlo renderings

Session/Category Title: Rendering Systems

Presenter(s)/Author(s):


Moderator(s):


Abstract:


    Regression-based algorithms have shown to be good at denoising Monte Carlo (MC) renderings by leveraging its inexpensive by-products (e.g., feature buffers). However, when using higher-order models to handle complex cases, these techniques often overfit to noise in the input. For this reason, supervised learning methods have been proposed that train on a large collection of reference examples, but they use explicit filters that limit their denoising ability. To address these problems, we propose a novel, supervised learning approach that allows the filtering kernel to be more complex and general by leveraging a deep convolutional neural network (CNN) architecture. In one embodiment of our framework, the CNN directly predicts the final denoised pixel value as a highly non-linear combination of the input features. In a second approach, we introduce a novel, kernel-prediction network which uses the CNN to estimate the local weighting kernels used to compute each denoised pixel from its neighbors. We train and evaluate our networks on production data and observe improvements over state-of-the-art MC denoisers, showing that our methods generalize well to a variety of scenes. We conclude by analyzing various components of our architecture and identify areas of further research in deep learning for MC denoising.

References:


    1. Martín Abadi, Ashish Agarwal, Paul Barham,, and others. 2015. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. (2015). http://tensorflow.org/Softwareavailablefromtensorflow.org.Google Scholar
    2. David Balduzzi, Brian McWilliams, and Tony Butler-Yeoman. 2016. Neural Taylor Approximations: Convergence and Exploration in Rectifier Networks. arXiv preprint arXiv:1611.02345 (2016).Google Scholar
    3. Pablo Bauszat, Martin Eisemann, and Marcus Magnor. 2011. Guided Image Filtering for Interactive High-quality Global Illumination. Computer Graphics Forum 30, 4 (2011), 1361–1368. Google ScholarDigital Library
    4. Benedikt Bitterli. 2016. Rendering Resources. (2016). https://benedikt-bitterli.me/resources/.Google Scholar
    5. Benedikt Bitterli, Fabrice Rousselle, Bochang Moon, José A. Iglesias-Guitián, David Adler, Kenny Mitchell, Wojciech Jarosz, and Jan Novák. 2016. Nonlinearly Weighted First-order Regression for Denoising Monte Carlo Renderings. Computer Graphics Forum 35, 4 (2016), 107–117. Google ScholarDigital Library
    6. Antoni Buades, Bartomeu Coll, and Jean-Michel Morel. 2005. A Review of Image Denoising Algorithms, with a New One. Multiscale Modeling & Simulation 4, 2 (2005), 490–530. Google ScholarCross Ref
    7. H. C. Burger, C.J. Schuler, and S. Harmeling. 2012. Image Denoising: Can Plain Neural Networks Compete with BM3D?. In 2012 IEEE Conference on Computer Vision and Pattern Recognition. 2392–2399. Google ScholarCross Ref
    8. Chakravarty R. A. Chaitanya, Anton Kaplanyan, Christoph Schied, Marco Salvi, Aaron Lefohn, Derek Nowrouzezahrai, and Timo Aila. 2017. Interactive Reconstruction of Noisy Monte Carlo Image Sequences using a Recurrent Autoencoder. ACM Trans. Graph. (Proc. SIGGRAPH) (2017).Google Scholar
    9. Robert L. Cook, Loren Carpenter, and Edwin Catmull. 1987. The Reyes Image Rendering Architecture. SIGGRAPH Comput. Graph. 21, 4 (Aug. 1987), 95–102. Google ScholarDigital Library
    10. Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. 2006. Image Denoising with Block-Matching and 3D Filtering. (2006).Google Scholar
    11. Jerome Friedman, Trevor Hastie, and Robert Tibshirani. 2001. The Elements of Statistical Learning. Vol. 1. Springer series in statistics Springer, Berlin.Google Scholar
    12. Michaël Gharbi, Gaurav Chaurasia, Sylvain Paris, and Frédo Durand. 2016. Deep Joint Demosaicking and Denoising. ACM Trans. Graph. 35, 6, Article 191 (Nov. 2016), 12 pages.Google ScholarDigital Library
    13. Xavier Glorot and Yoshua Bengio. 2010. Understanding the Difficulty of Training Deep Feedforward Neural Networks. In International conference on artificial intelligence and statistics. 249–256.Google Scholar
    14. Luke Goddard. 2014. Silencing the Noise on Elysium. In ACM SIGGRAPH 2014 Talks (SIGGRAPH ’14). ACM, New York, NY, USA, Article 38, 1 pages.Google Scholar
    15. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). http://arxiv.org/abs/1512.03385Google Scholar
    16. James T. Kajiya. 1986. The Rendering Equation. SIGGRAPH Comput. Graph. 20, 4 (Aug. 1986), 143–150. Google ScholarDigital Library
    17. Nima Khademi Kalantari, Steve Bako, and Pradeep Sen. 2015. A Machine Learning Approach for Filtering Monte Carlo Noise. 34, 4, Article 122 (July 2015), 12 pages.Google Scholar
    18. Nima Khademi Kalantari and Pradeep Sen. 2013. Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms. 32, 2pt1 (2013), 93–102.Google Scholar
    19. A. Keller, L. Fascione, M. Fajardo, I. Georgiev, P. Christensen, J. Hanika, C. Eisenacher, and G. Nichols. 2015. The Path Tracing Revolution in the Movie Industry. In ACM SIGGRAPH 2015 Courses (SIGGRAPH ’15). ACM, New York, NY, USA, Article 24, 7 pages. Google ScholarDigital Library
    20. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. CoRR abs/1412.6980 (2014). http://arxiv.org/abs/1412.6980Google Scholar
    21. Diederik P Kingma and Max Welling. 2013. Auto-Encoding Variational Bayes. In International Conference on Learning Representations.Google Scholar
    22. Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep Learning. Nature 521 (2015), 436–444. Google ScholarCross Ref
    23. Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and others. 2016. Photo-Realistic Single Image Super-Resolution using a Generative Adversarial Network. arXiv preprint arXiv:1609.04802 (2016).Google Scholar
    24. Tzu-Mao Li, Yu-Ting Wu, and Yung-Yu Chuang. 2012. SURE-based Optimization for Adaptive Sampling and Reconstruction. ACM Trans. Graph. 31, 6, Article 194 (Nov. 2012), 9 pages.Google ScholarDigital Library
    25. Michael D. McCool. 1999. Anisotropic Diffusion for Monte Carlo Noise Reduction. ACM Transactions on Graphics 18, 2 (April 1999), 171–194. Google ScholarDigital Library
    26. Bochang Moon, Nathan Carr, and Sung-Eui Yoon. 2014. Adaptive Rendering Based on Weighted Local Regression. ACM Trans. Graph. 33, 5 (Sept. 2014), 170:1–170:14.Google ScholarDigital Library
    27. Bochang Moon, Jong Yun Jun, JongHyeob Lee, Kunho Kim, Toshiya Hachisuka, and Sung-Eui Yoon. 2013. Robust Image Denoising Using a Virtual Flash Image for Monte Carlo Ray Tracing. Computer Graphics Forum 32, 1 (2013), 139–151. Google ScholarCross Ref
    28. Bochang Moon, Steven McDonagh, Kenny Mitchell, and Markus Gross. 2016. Adaptive Polynomial Rendering. To appear in ACM Trans. Graph. (Proc. SIGGRAPH) (2016), 10.Google Scholar
    29. Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. 2016. Wavenet: A Generative Model for Raw Audio. arXiv preprint arXiv:1609.03499 (2016).Google Scholar
    30. Fabrice Rousselle, Claude Knaus, and Matthias Zwicker. 2011. Adaptive Sampling and Reconstruction using Greedy Error Minimization. ACM Trans. Graph. 30, 6, Article 159 (Dec. 2011), 12 pages. Google ScholarDigital Library
    31. Fabrice Rousselle, Claude Knaus, and Matthias Zwicker. 2012. Adaptive Rendering with Non-local Means Filtering. 31, 6, Article 195 (Nov. 2012), 11 pages.Google Scholar
    32. Fabrice Rousselle, Marco Manzi, and Matthias Zwicker. 2013. Robust Denoising using Feature and Color Information. Computer Graphics Forum 32, 7 (2013), 121–130. Google ScholarCross Ref
    33. Holly E. Rushmeier and Gregory J. Ward. 1994. Energy Preserving Non-Linear Filters. In Proc. 21st annual Conf. on Computer graphics and interactive techniques (SIGGRAPH ’94). ACM, 131–138. Google ScholarDigital Library
    34. Tim Salimans and Diederik P Kingma. 2016. Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks. In Adv in Neural Information Processing Systems (NIPS).Google Scholar
    35. Pradeep Sen and Soheil Darabi. 2012. On Filtering the Noise from the Random Parameters in Monte Carlo Rendering. ACM Transactions on Graphics 31, 3, Article 18 (June 2012), 15 pages.Google ScholarDigital Library
    36. Pradeep Sen, Matthias Zwicker, Fabrice Rousselle, Sung-Eui Yoon, and Nima Khademi Kalantari. 2015. Denoising Your Monte Carlo Renders: Recent Advances in Image-space Adaptive Sampling and Reconstruction. In ACM SIGGRAPH 2015 Courses. ACM, 11. Google ScholarDigital Library
    37. Karen Simonyan and Andrew Zisserman. 2014. Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv preprint arXiv:1409.1556 (2014).Google Scholar
    38. Charles M. Stein. 1981. Estimation of the Mean of a Multivariate Normal Distribution. The Annals of Statistics 9, 6 (1981), 1135–1151. http://www.jstor.org/stable/2240405 Google ScholarCross Ref
    39. Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. 2004. Image Quality Assessment: from Error Visibility to Structural Similarity. IEEE Transactions on Image Processing 13, 4 (April 2004), 600–612. Google ScholarDigital Library
    40. Junyuan Xie, Linli Xu, and Enhong Chen. 2012. Image Denoising and Inpainting with Deep Neural Networks. In Advances in Neural Information Processing Systems. 341–349.Google Scholar
    41. Wenhan Yang, Jiashi Feng, Jianchao Yang, Fang Zhao, Jiaying Liu, Zongming Guo, and Shuicheng Yan. 2016. Deep Edge Guided Recurrent Residual Learning for Image Super-Resolution. CoRR abs/1604.08671 (2016). http://arxiv.org/abs/1604.08671Google Scholar
    42. Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. 2016. Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising. arXiv preprint arXiv:1608.03981 (2016).Google Scholar
    43. Henning Zimmer, Fabrice Rousselle, Wenzel Jakob, Oliver Wang, David Adler, Wojciech Jarosz, Olga Sorkine-Hornung, and Alexander Sorkine-Hornung. 2015. Path-space Motion Estimation and Decomposition for Robust Animation Filtering. Computer Graphics Forum 34, 4 (2015), 131–142. Google ScholarDigital Library
    44. Matthias Zwicker, Wojciech Jarosz, Jaakko Lehtinen, Bochang Moon, Ravi Ramamoorthi, Fabrice Rousselle, Pradeep Sen, Cyril Soler, and Sung-Eui Yoon. 2015. Recent Advances in Adaptive Sampling and Reconstruction for Monte Carlo Rendering. 34, 2 (May 2015), 667–681.Google Scholar

ACM Digital Library Publication:


Overview Page: