“Sample-based Monte Carlo denoising using a kernel-splatting network” by Gharbi, Li, Aittala, Lehtinen and Durand

  • ©Michaël Gharbi, Tzu-Mao Li, Miika Aittala, Jaakko Lehtinen, and Frédo Durand



Session Title:

    Machine Learning for Rendering


    Sample-based Monte Carlo denoising using a kernel-splatting network



    Denoising has proven to be useful to efficiently generate high-quality Monte Carlo renderings. Traditional pixel-based denoisers exploit summary statistics of a pixel’s sample distributions, which discards much of the samples’ information and limits their denoising power. On the other hand, sample-based techniques tend to be slow and have difficulties handling general transport scenarios. We present the first convolutional network that can learn to denoise Monte Carlo renderings directly from the samples. Learning the mapping between samples and images creates new challenges for the network architecture design: the order of the samples is arbitrary, and they should be treated in a permutation invariant manner. To address these challenges, we develop a novel kernel-predicting architecture that splats individual samples onto nearby pixels. Splatting is a natural solution to situations such as motion blur, depth-of-field and many light transport paths, where it is easier to predict which pixels a sample contributes to, rather than a gather approach that needs to figure out, for each pixel, which samples (or nearby pixels) are relevant. Compared to previous state-of-the-art methods, ours is robust to the severe noise of low-sample count images (e.g. 8 samples per pixel) and yields higher-quality results both visually and numerically. Our approach retains the generality and efficiency of pixel-space methods while enjoying the expressiveness and accuracy of the more complex sample-based approaches.


    1. Miika Aittala and Frédo Durand. 2018. Burst image deblurring using permutation invariant convolutional neural networks. ECCV (2018).Google Scholar
    2. Steve Bako, Thijs Vogels, Brian Mcwilliams, Mark Meyer, Jan Novák, Alex Harvill, Pradeep Sen, Tony Derose, and Fabrice Rousselle. 2017. Kernel-predicting convolutional networks for denoising Monte Carlo renderings. ACM SIGGRAPH (2017). Google ScholarDigital Library
    3. Pablo Bauszat, Martin Eisemann, S John, and M Magnor. 2015. Sample-based manifold filtering for interactive global illumination and depth of field. Computer Graphics Forum (2015). Google ScholarDigital Library
    4. Pablo Bauszat, Martin Eisemann, and Marcus Magnor. 2011. Guided Image Filtering for Interactive High-quality Global Illumination. Computer Graphics Forum (Proc. EGSR) (2011). Google ScholarDigital Library
    5. Laurent Belcour, Cyril Soler, Kartic Subr, Nicolas Holzschuch, and Fredo Durand. 2013. 5D covariance tracing for efficient defocus and motion blur. ACM TOG (2013). Google ScholarDigital Library
    6. Benedikt Bitterli. 2016. Rendering resources. https://benedikt-bitterli.me/resources.Google Scholar
    7. Benedikt Bitterli, Fabrice Rousselle, Bochang Moon, José A. Iglesias-Guitiàn, David Adler, Kenny Mitchell, Wojciech Jarosz, and Jan Novák. 2016. Nonlinearly Weighted First-order Regression for Denoising Monte Carlo Renderings. Computer Graphics Forum (Proc. EGSR) (2016).Google Scholar
    8. Nicolas Bonneel, James Tompkin, Kalyan Sunkavalli, Deqing Sun, Sylvain Paris, and Hanspeter Pfister. 2015. Blind Video Temporal Consistency. ACM SIGGRAPH Asia (2015). Google ScholarDigital Library
    9. Chakravarty R. Alla Chaitanya, Anton S. Kaplanyan, Christoph Schied, Marco Salvi, Aaron Lefohn, Derek Nowrouzezahrai, and Timo Aila. 2017. Interactive Reconstruction of Monte Carlo Image Sequences Using a Recurrent Denoising Autoencoder. ACM SIGGRAPH (2017).Google Scholar
    10. Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. 2015. ShapeNet: An Information-Rich 3D Model Repository. CoRR (2015).Google Scholar
    11. M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. 2014. Describing Textures in the Wild. CVPR (2014). Google ScholarDigital Library
    12. Mauricio Delbracio, Pablo Musé, Antoni Buades, Julien Chauvier, Nicholas Phelps, and Jean-Michel Morel. 2014. Boosting Monte Carlo rendering by ray histogram fusion. ACM TOG (2014). Google ScholarDigital Library
    13. Frédo Durand, Nicolas Holzschuch, Cyril Soler, Eric Chan, and François X Sillion. 2005. A frequency analysis of light transport. ACM SIGGRAPH (2005). Google ScholarDigital Library
    14. Kevin Egan, Florian Hecht, Frédo Durand, and Ravi Ramamoorthi. 2011. Frequency analysis and sheared filtering for shadow light fields of complex occluders. ACM TOG (2011). Google ScholarDigital Library
    15. Kevin Egan, Yu-Ting Tseng, Nicolas Holzschuch, Frédo Durand, and Ravi Ramamoorthi. 2009. Frequency analysis and sheared reconstruction for rendering motion blur. ACM SIGGRAPH (2009). Google ScholarDigital Library
    16. Eduardo SL Gastal and Manuel M Oliveira. 2012. Adaptive manifolds for real-time high-dimensional filtering. ACM SIGGRAPH (2012).Google Scholar
    17. Toshiya Hachisuka, Wojciech Jarosz, Richard Peter Weistroffer, Kevin Dale, Greg Humphreys, Matthias Zwicker, and Henrik Wann Jensen. 2008. Multidimensional adaptive sampling and reconstruction for ray tracing. ACM SIGGRAPH (2008). Google ScholarDigital Library
    18. Kaiming He, Jian Sun, and Xiaoou Tang. 2010. Guided image filtering. ECCV (2010).Google Scholar
    19. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. ICCV (2015). Google ScholarDigital Library
    20. James T. Kajiya. 1986. The Rendering Equation. ACM SIGGRAPH (1986). Google ScholarDigital Library
    21. Nima Khademi Kalantari, Steve Bako, and Pradeep Sen. 2015. A Machine Learning Approach for Filtering Monte Carlo Noise. ACM SIGGRAPH (2015).Google Scholar
    22. Nima Khademi Kalantari and Pradeep Sen. 2013. Removing the noise in Monte Carlo rendering with general image denoising algorithms. Computer Graphics Forum (Proc. EG) (2013).Google Scholar
    23. Diederick P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. International Conference on Learning Representations (2015).Google Scholar
    24. Thomas Kollig and Alexander Keller. 2002. Efficient multidimensional sampling. Computer Graphics Forum (Proc. EG) (2002).Google Scholar
    25. Jaakko Lehtinen, Timo Aila, Jiawen Chen, Samuli Laine, and Frédo Durand. 2011. Temporal light field reconstruction for rendering distribution effects. ACM SIGGRAPH (2011). Google ScholarDigital Library
    26. Jaakko Lehtinen, Timo Aila, Samuli Laine, and Frédo Durand. 2012. Reconstructing the indirect light field for global illumination. ACM SIGGRAPH (2012). Google ScholarDigital Library
    27. Thomas Leimkühler, Hans-Peter Seidel, and Tobias Ritschel. 2018. Laplacian kernel splatting for efficient depth-of-field and motion blur synthesis or reconstruction. ACM SIGGRAPH (2018).Google Scholar
    28. Tzu-Mao Li, Michaël Gharbi, Andrew Adams, Frédo Durand, and Jonathan Ragan-Kelley. 2018. Differentiable programming for image processing and deep learning in Halide. ACM SIGGRAPH (2018). Google ScholarDigital Library
    29. Tzu-Mao Li, Yu-Ting Wu, and Yung-Yu Chuang. 2012. SURE-based optimization for adaptive sampling and reconstruction. ACM SIGGRAPH Asia (2012). Google ScholarDigital Library
    30. Michael Mara, Morgan McGuire, Benedikt Bitterli, and Wojciech Jarosz. 2017. An Efficient Denoising Algorithm for Global Illumination. High Performance Graphics (2017). Google ScholarDigital Library
    31. Michael D McCool. 1999. Anisotropic diffusion for Monte Carlo noise reduction. ACM TOG (1999). Google ScholarDigital Library
    32. Soham Uday Mehta, JiaXian Yao, Ravi Ramamoorthi, and Fredo Durand. 2014. Factored Axis-aligned Filtering for Rendering Multiple Distribution Effects. ACM SIGGRAPH (2014).Google Scholar
    33. Bochang Moon, Nathan Carr, and Sung-Eui Yoon. 2014. Adaptive rendering based on weighted local regression. ACM TOG (2014). Google ScholarDigital Library
    34. Ryan S Overbeck, Craig Donner, and Ravi Ramamoorthi. 2009. Adaptive wavelet rendering. ACM SIGGRAPH Asia (2009). Google ScholarDigital Library
    35. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch. (2017).Google Scholar
    36. Matt Pharr and Greg Humphreys. 2010. Physically based rendering: from theory to implementation. Google ScholarDigital Library
    37. Matt Pharr, Wenzel Jakob, and Greg Humphreys. 2016. Physically based rendering: From theory to implementation. Google ScholarDigital Library
    38. Charles Ruizhongtai Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. 2017. PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation. CVPR (2017).Google Scholar
    39. Jonathan Ragan-Kelley, Andrew Adams, Sylvain Paris, Marc Levoy, Saman Amarasinghe, and Frédo Durand. 2012. Decoupling Algorithms from Schedules for Easy Optimization of Image Processing Pipelines. ACM SIGGRAPH (2012). Google ScholarDigital Library
    40. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. International Conference on Medical Image Computing and Computer-Assisted Intervention (2015).Google ScholarCross Ref
    41. Fabrice Rousselle, Claude Knaus, and Matthias Zwicker. 2012. Adaptive rendering with non-local means filtering. ACM SIGGRAPH Asia (2012). Google ScholarDigital Library
    42. Fabrice Rousselle, Marco Manzi, and Matthias Zwicker. 2013. Robust denoising using feature and color information. Computer Graphics Forum (Proc. PG) (2013).Google Scholar
    43. Holly E. Rushmeier and Gregory J. Ward. 1994. Energy Preserving Non-linear Filters. ACM SIGGRAPH (1994). Google ScholarDigital Library
    44. Christoph Schied, Anton Kaplanyan, Chris Wyman, Anjul Patney, Chakravarty R. Alla Chaitanya, John Burgess, Shiqiu Liu, Carsten Dachsbacher, Aaron Lefohn, and Marco Salvi. 2017. Spatiotemporal Variance-guided Filtering: Real-time Reconstruction for Path-traced Global Illumination. High Performance Graphics (2017). Google ScholarDigital Library
    45. Pradeep Sen and Soheil Darabi. 2012. On filtering the noise from the random parameters in Monte Carlo rendering. ACM TOG (2012). Google ScholarDigital Library
    46. Shuran Song, Fisher Yu, Andy Zeng, Angel X Chang, Manolis Savva, and Thomas Funkhouser. 2017. Semantic Scene Completion from a Single Depth Image. CVPR (2017).Google Scholar
    47. Thijs Vogels, Fabrice Rousselle, Brian McWilliams, Gerhard Röthlin, Alex Harvill, David Adler, Mark Meyer, and Jan Novák. 2018. Denoising with kernel prediction and asymmetric loss functions. ACM SIGGRAPH (2018). Google ScholarDigital Library
    48. Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE TIP (2004). Google ScholarDigital Library
    49. Gregory J. Ward, Francis M. Rubinstein, and Robert D. Clear. 1988. A Ray Tracing Solution for Diffuse Interreflection. ACM SIGGRAPH (1988). Google ScholarDigital Library
    50. Greg Zaal. 2016. HDRI Haven. https://hdrihaven.com.Google Scholar
    51. Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan R Salakhutdinov, and Alexander J Smola. 2017. Deep sets. NIPS (2017). Google ScholarDigital Library
    52. Henning Zimmer, Fabrice Rousselle, Wenzel Jakob, Oliver Wang, David Adler, Wojciech Jarosz, Olga Sorkine-Hornung, and Alexander Sorkine-Hornung. 2015. Path-space Motion Estimation and Decomposition for Robust Animation Filtering. Computer Graphics Forum (Proc. EGSR) (2015).Google Scholar
    53. M. Zwicker, W. Jarosz, J. Lehtinen, B. Moon, R. Ramamoorthi, F. Rousselle, P. Sen, C. Soler, and S.-E. Yoon. 2015. Recent Advances in Adaptive Sampling and Reconstruction for Monte Carlo Rendering. Computer Graphics Forum (Proc. EG) (2015). Google ScholarDigital Library

ACM Digital Library Publication:

Overview Page: