“Denoising with kernel prediction and asymmetric loss functions” by Vogels, Rousselle, Mcwilliams, Rothlin, Harvill, et al. …

  • ©Thijs Vogels, Fabrice Rousselle, Brian Mcwilliams, Gerhard Röthlin, Alex Harvill, David Adler, Mark Meyer, and Jan Novák

Conference:


Type:


Entry Number: 124

Title:

    Denoising with kernel prediction and asymmetric loss functions

Session/Category Title: Learning for Rendering and Material Acquisition

Presenter(s)/Author(s):


Moderator(s):


Abstract:


    We present a modular convolutional architecture for denoising rendered images. We expand on the capabilities of kernel-predicting networks by combining them with a number of task-specific modules, and optimizing the assembly using an asymmetric loss. The source-aware encoder—the first module in the assembly—extracts low-level features and embeds them into a common feature space, enabling quick adaptation of a trained network to novel data. The spatial and temporal modules extract abstract, high-level features for kernel-based reconstruction, which is performed at three different spatial scales to reduce low-frequency artifacts. The complete network is trained using a class of asymmetric loss functions that are designed to preserve details and provide the user with a direct control over the variance-bias trade-off during inference. We also propose an error-predicting module for inferring reconstruction error maps that can be used to drive adaptive sampling. Finally, we present a theoretical analysis of convergence rates of kernel-predicting architectures, shedding light on why kernel prediction performs better than synthesizing the colors directly, complementing the empirical evidence presented in this and previous works. We demonstrate that our networks attain results that compare favorably to state-of-the-art methods in terms of detail preservation, low-frequency noise removal, and temporal stability on a variety of production and academic datasets.

References:


    1. Martín Abadi, Ashish Agarwal, Paul Barham, et al. 2015. TensorFlow: Large-scale machine learning on heterogeneous systems, http://tensorflow.org/Google Scholar
    2. Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016. Neural module networks. In IEEE Conference on Computer Vision and Pattern Recognition (June 27–30). 39–48.Google ScholarCross Ref
    3. Steve Bako, Thijs Vogels, Brian Mcwilliams, Mark Meyer, Jan Novák, Alex Harvill, Pradeep Sen, Tony Derose, and Fabrice Rousselle. 2017. Kernel-predicting convolutional networks for denoising Monte Carlo renderings. ACM Trans. Graphics (Proc. SIGGRAPH) 36, 4, Article 97 (July 2017), 14 pages. Google ScholarDigital Library
    4. David Balduzzi, Brian McWilliams, and Tony Butler-Yeoman. 2017. Neural Taylor approximations: Convergence and exploration in rectifier networks. In Proc. 34th International Conference on Machine Learning (Proc. Machine Learning Research), Vol. 70. PMLR, Sydney, Australia, 351–360.Google Scholar
    5. Pablo Bauszat, Martin Eisemann, and Marcus Magnor. 2011. Guided image filtering for interactive high-quality global illumination. Computer Graphics Forum 30, 4 (2011), 1361–1368. Google ScholarDigital Library
    6. Amir Beck and Marc Teboulle. 2003. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters 31, 3 (2003), 167–175. Google ScholarDigital Library
    7. Benedikt Bitterli, Fabrice Rousselle, Bochang Moon, José A. Iglesias-Guitián, David Adler, Kenny Mitchell, Wojciech Jarosz, and Jan Novák. 2016. Nonlinearly weighted first-order regression for denoising Monte Carlo renderings. Computer Graphics Forum 35, 4 (2016), 107–117. Google ScholarDigital Library
    8. Mark R. Bolin and Gary W. Meyer. 1998. A perceptually based adaptive sampling algorithm. In Proc. 25th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’98). ACM Press, New York, NY, USA, 299–309. Google ScholarDigital Library
    9. Antoni Buades, Bartomeu Coll, and Jean-Michel Morel. 2005. A review of image denoising algorithms, with a new one. Multiscale Modeling & Simulation 4, 2 (2005), 490–530.Google ScholarCross Ref
    10. Antoni Buades, Bartomeu Coll, and Jean-Michel Morel. 2008. Nonlocal image and movie denoising. International Journal of Computer Vision 7 6, 2 (01 Feb 2008), 123–139. Google ScholarDigital Library
    11. Sébastien Bubeck. 2015. Convex optimization: Algorithms and complexity. Foundations and Trends® in Machine Learning 8, 3–4 (2015), 231–358. Google ScholarDigital Library
    12. Harold Christopher Burger, Christian J. Schuler, and Stefan Harmeling. 2012. Image denoising: Can plain neural networks compete with BM3D?. In IEEE Conference on Computer Vision and Pattern Recognition (June 16–21). IEEE Computer Society, 2392–2399. Google ScholarDigital Library
    13. Chakravarty R. Alla Chaitanya, Anton S. Kaplanyan, Christoph Schied, Marco Salvi, Aaron Lefohn, Derek Nowrouzezahrai, and Timo Aila. 2017. Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder. ACM Trans. Graphics (Proc. SIGGRAPH) 36, 4, Article 98 (July 2017), 12 pages. Google ScholarDigital Library
    14. Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. 2006. Image denoising with block-matching and 3D filtering. In Proc. SPIE, Vol. 6064. 606414-606414-12.Google ScholarCross Ref
    15. Mauricio Delbracio, Pablo Musé, Antoni Buades, Julien Chauvier, Nicholas Phelps, and Jean-Michel Morel. 2014. Boosting Monte Carlo rendering by ray histogram fusion. ACM Trans. Graphics 33, 1, Article 8 (Feb. 2014), 15 pages. Google ScholarDigital Library
    16. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research 12, Jul (2011), 2121–2159. Google ScholarDigital Library
    17. Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proc. 13th International Conference on Artificial Intelligence and Statistics (May 13–15). JMLR.org, 249–256.Google Scholar
    18. Toshiya Hachisuka, Wojciech Jarosz, Richard Peter Weistroffer, Kevin Dale, Greg Humphreys, Matthias Zwicker, and Henrik Wann Jensen. 2008. Multidimensional adaptive sampling and reconstruction for ray tracing. ACM Trans. Graphics 27, 3, Article 33 (Aug. 2008), 10 pages. Google ScholarDigital Library
    19. Moritz Hardt, Ben Recht, and Yoram Singer. 2016. Train faster, generalize better: Stability of stochastic gradient descent. In Proc. 33nd International Conference on Machine Learning (June 19–24). JMLR.org, 1225–1234. Google ScholarDigital Library
    20. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition (June 27–30). IEEE Computer Society, 770–778.Google ScholarCross Ref
    21. Viren Jain and H. Sebastian Seung. 2008. Natural image denoising with convolutional networks. In Advances in Neural Information Processing Systems 21 (December 8–11). Curran Associates, Inc., 769–776. Google ScholarDigital Library
    22. Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Luc Van Gool. 2016. Dynamic filter networks. In Advances in Neural Information Processing Systems 29 (December 5–10). Curran Associates, Inc., 667–675. Google ScholarDigital Library
    23. Nima Khademi Kalantari, Steve Bako, and Pradeep Sen. 2015. A machine learning approach for filtering Monte Carlo noise. ACM Trans. Graphics (Proc. SIGGRAPH) 34, 4, Article 122 (July 2015), 12 pages. Google ScholarDigital Library
    24. Alexander Keller, Luca Fascione, Marcos Fajardo, Iliyan Georgiev, Per H. Christensen, Johannes Hanika, Christian Eisenacher, and Gregory Nichols. 2015. The path tracing revolution in the movie industry. In ACM SIGGRAPH 2015 Courses (SIGGRAPH ’15). ACM Press, New York, NY, USA, Article 24, 7 pages. Google ScholarDigital Library
    25. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. CoRR abs/1412.6980 (2014). arXiv:1412.6980Google Scholar
    26. James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. 2017. Overcoming catastrophic forgetting in neural networks. In Proc. National Academy of Sciences, Vol. 114. National Academy of Sciences, 3521–3526.Google ScholarCross Ref
    27. Gabriel Krummenacher, Brian McWilliams, Yannic Kilcher, Joachim M. Buhmann, and Nicolai Meinshausen. 2016. Scalable adaptive stochastic optimization using random projections. In Advances in Neural Information Processing Systems 29 (December 5–10). Curran Associates, Inc., 1750–1758. Google ScholarDigital Library
    28. Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. Nature 521 (2015), 436–444.Google ScholarCross Ref
    29. Tzu-Mao Li, Yu-Ting Wu, and Yung-Yu Chuang. 2012. SURE-based optimization for adaptive sampling and reconstruction. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 31, 6, Article 194 (Nov. 2012), 9 pages. Google ScholarDigital Library
    30. Aurélien Lucchi, Brian McWilliams, and Thomas Hofmann. 2015. A variance reduced stochastic Newton method. CoRR abs/1503.08316 (2015). arXiv:1503.08316Google Scholar
    31. Michael D. McCool. 1999. Anisotropic diffusion for Monte Carlo noise reduction. ACM Trans. Graphics 18, 2 (April 1999), 171–194. Google ScholarDigital Library
    32. Ben Mildenhall, Jonathan T. Barron, Jiawen Chen, Dillon Sharlet, Ren Ng, and Robert Carroll. 2017. Burst denoising with kernel prediction networks. CoRR abs/1712.02327 (2017). arXiv:1712.02327Google Scholar
    33. Don P. Mitchell. 1987. Generating antialiased images at low sampling densities. In Proc. 14th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’87). ACM Press, New York, NY, USA, 65–72. Google ScholarDigital Library
    34. Bochang Moon, Nathan Carr, and Sung-Eui Yoon. 2014. Adaptive rendering based on weighted local regression. ACM Trans. Graphics 33, 5, Article 170 (Sept. 2014), 14 pages. Google ScholarDigital Library
    35. Bochang Moon, Jong Yun Jun, JongHyeob Lee, Kunho Kim, Toshiya Hachisuka, and Sung-Eui Yoon. 2013. Robust image denoising using a virtual flash image for Monte Carlo ray tracing. Computer Graphics Forum 32, 1 (2013), 139–151.Google ScholarCross Ref
    36. Bochang Moon, Steven McDonagh, Kenny Mitchell, and Markus Gross. 2016. Adaptive polynomial rendering. ACM Trans. Graphics (Proc. SIGGRAPH) 35, 4, Article 40 (July 2016), 10 pages. Google ScholarDigital Library
    37. Kevin P. Murphy. 2012. Machine learning – a probabilistic perspective. MIT Press. Google ScholarDigital Library
    38. Simon Niklaus, Long Mai, and Feng Liu. 2017. Video frame interpolation via adaptive convolution. In IEEE Conference on Computer Vision and Pattern Recognition (July 21–26). IEEE Computer Society, 2270–2279.Google Scholar
    39. Ryan S. Overbeck, Craig Donner, and Ravi Ramamoorthi. 2009. Adaptive wavelet rendering. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 28, 5, Article 140 (Dec. 2009), 12 pages. Google ScholarDigital Library
    40. Mahesh Ramasubramanian, Sumanta N. Pattanaik, and Donald P. Greenberg. 1999. A perceptually based physical error metric for realistic image synthesis. In Proc. 26th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’99). ACM Press/Addison-Wesley Publishing Co., New York, NY, USA, 73–82. Google ScholarDigital Library
    41. Garvesh Raskutti and Sayan Mukherjee. 2015. The information geometry of mirror descent. IEEE Trans. Information Theory 61, 3 (2015), 1451–1457.Google ScholarDigital Library
    42. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-Net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention – 18th International Conf. (October 5–9). Springer, 234–241.Google Scholar
    43. Fabrice Rousselle, Claude Knaus, and Matthias Zwicker. 2011. Adaptive sampling and reconstruction using greedy error minimization. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 30, 6, Article 159 (Dec. 2011), 12 pages. Google ScholarDigital Library
    44. Fabrice Rousselle, Claude Knaus, and Matthias Zwicker. 2012. Adaptive rendering with non-local means filtering. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 31, 6, Article 195 (Nov. 2012), 11 pages. Google ScholarDigital Library
    45. Fabrice Rousselle, Marco Manzi, and Matthias Zwicker. 2013. Robust denoising using feature and color information. Computer Graphics Forum 32, 7 (2013), 121–130.Google ScholarCross Ref
    46. Holly E. Rushmeier and Gregory J. Ward. 1994. Energy preserving non-linear filters. In Proc. 21st Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’94). ACM Press, 131–138. Google ScholarDigital Library
    47. Pradeep Sen and Soheil Darabi. 2012. On filtering the noise from the random parameters in Monte Carlo rendering. ACM Trans. Graphics 31, 3, Article 18 (June 2012), 15 pages. Google ScholarDigital Library
    48. Shai Shalev-Shwartz et al. 2012. Online learning and online convex optimization. Foundations and Trends® in Machine Learning 4, 2 (2012), 107–194. Google ScholarDigital Library
    49. Thijs Vogels. 2016. Kernel-predicting convolutional networks for denoising Monte Carlo renderings. Master’s thesis. ETH Zürich, Zürich, Switzerland.Google Scholar
    50. Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. 2004. Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing 13, 4 (April 2004), 600–612. Google ScholarDigital Library
    51. Ashia C. Wilson, Rebecca Roelofs, Mitchell Stern, Nati Srebro, and Benjamin Recht. 2017. The marginal value of adaptive gradient methods in machine learning. In Advances in Neural Information Processing Systems 30 (December 4–9). Curran Associates, Inc., 4151–4161.Google Scholar
    52. Matthias Zwicker, Wojciech Jarosz, Jaakko Lehtinen, Bochang Moon, Ravi Ramamoorthi, Fabrice Rousselle, Pradeep Sen, Cyril Soler, and Sung-Eui Yoon. 2015. Recent advances in adaptive sampling and reconstruction for Monte Carlo rendering. Computer Graphics Forum (Proc. Eurographics) 34, 2 (May 2015), 667–681. Google ScholarDigital Library

ACM Digital Library Publication:


Overview Page: