“Deep convolutional reconstruction for gradient-domain rendering” by Kettunen, Härkönen and Lehtinen
Conference:
Type:
Title:
- Deep convolutional reconstruction for gradient-domain rendering
Session/Category Title: Machine Learning for Rendering
Presenter(s)/Author(s):
Abstract:
It has been shown that rendering in the gradient domain, i.e., estimating finite difference gradients of image intensity using correlated samples, and combining them with direct estimates of pixel intensities by solving a screened Poisson problem, often offers fundamental benefits over merely sampling pixel intensities. The reasons can be traced to the frequency content of the light transport integrand and its interplay with the gradient operator. However, while they often yield state of the art performance among algorithms that are based on Monte Carlo sampling alone, gradient-domain rendering algorithms have, until now, not generally been competitive with techniques that combine Monte Carlo sampling with post-hoc noise removal using sophisticated non-linear filtering.Drawing on the power of modern convolutional neural networks, we propose a novel reconstruction method for gradient-domain rendering. Our technique replaces the screened Poisson solver of previous gradient-domain techniques with a novel dense variant of the U-Net autoencoder, additionally taking auxiliary feature buffers as inputs. We optimize our network to minimize a perceptual image distance metric calibrated to the human visual system. Our results significantly improve the quality obtained from gradient-domain path tracing, allowing it to overtake state-of-the-art comparison techniques that denoise traditional Monte Carlo samplings. In particular, we observe that the correlated gradient samples — that offer information about the smoothness of the integrand unavailable in standard Monte Carlo sampling — notably improve image quality compared to an equally powerful neural model that does not make use of gradient samples.
References:
1. Jonghee Back, Sung-Eui Yoon, and Bochang Moon. 2018. Feature Generation for Adaptive Gradient-Domain Path Tracing. Computer Graphics Forum (2018).Google Scholar
2. Steve Bako, Thijs Vogels, Brian McWilliams, Mark Meyer, Jan Novák, Alex Harvill, Pradeep Sen, Tony DeRose, and Fabrice Rousselle. 2017. Kernel-Predicting Convolutional Networks for Denoising Monte Carlo Renderings. ACM Transactions on Graphics (Proceedings of SIGGRAPH 2017) 36, 4, Article 97 (2017), 97:1–97:14 pages. Google ScholarDigital Library
3. Pablo Bauszat, Victor Petitjean, and Elmar Eisemann. 2017. Gradient-domain Path Reusing. ACM Trans. Graph. 36, 6, Article 229 (Nov. 2017), 9 pages. Google ScholarDigital Library
4. Philippe Bekaert, Mateu Sbert, and John Halton. 2002. Accelerating Path Tracing by Reusing Paths. In Proceedings of the 13th Eurographics Workshop on Rendering (EGRW ’02). Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, 125–134. http://dl.acm.org/citation.cfm?id=581896.581914 Google ScholarDigital Library
5. Benedikt Bitterli and Wojciech Jarosz. 2017. Beyond Points and Beams: Higher-Dimensional Photon Samples for Volumetric Light Transport. ACM Transactions on Graphics (Proceedings of SIGGRAPH) 36, 4 (July 2017). Google ScholarDigital Library
6. Benedikt Bitterli, Fabrice Rousselle, Bochang Moon, José A. Iglesias-Guitián, David Adler, Kenny Mitchell, Wojciech Jarosz, and Jan Novák. 2016. Nonlinearly Weighted First-order Regression for Denoising Monte Carlo Renderings. In Proc. Eurographics Symposium on Rendering (EGSR) 2016.Google ScholarCross Ref
7. Antoni Buades, Bartomeu Coll, and Jean-Michel Morel. 2005. A review of image denoising algorithms, with a new one. SIAM Journal on Multiscale Modeling and Simulation: A SIAM Interdisciplinary Journal 4, 2 (2005), 490–530. https://hal.archives-ouvertes.fr/hal-00271141Google ScholarCross Ref
8. Chakravarty R. Alla Chaitanya, Anton S. Kaplanyan, Christoph Schied, Marco Salvi, Aaron Lefohn, Derek Nowrouzezahrai, and Timo Aila. 2017. Interactive Reconstruction of Monte Carlo Image Sequences Using a Recurrent Denoising Autoencoder. ACM Trans. Graph. 36, 4, Article 98 (July 2017), 12 pages. Google ScholarDigital Library
9. Iliyan Georgiev, Jaroslav Křivánek, Tomáš Davidovič, and Philipp Slusallek. 2012. Light Transport Simulation with Vertex Connection and Merging. ACM Trans. Graph. 31, 6, Article 192 (Nov. 2012), 10 pages. Google ScholarDigital Library
10. Adrien Gruson, Binh-Son Hua, Nicolas Vibert, Derek Nowrouzezahrai, and Toshiya Hachisuka. 2018. Gradient-domain Volumetric Photon Density Estimation. ACM Trans. Graph. 37, 4, Article 82 (July 2018), 13 pages. Google ScholarDigital Library
11. Steven Guan, Amir A. Khan, Siddhartha Sikdar, and Parag V. Chitnis. 2018. Fully Dense UNet for 2D Sparse Photoacoustic Tomography Artifact Removal. CoRR abs/1808.10848 (2018). arXiv:1808.10848 http://arxiv.org/abs/1808.10848Google Scholar
12. Toshiya Hachisuka and Henrik Wann Jensen. 2009. Stochastic Progressive Photon Mapping. In ACM SIGGRAPH Asia 2009 Papers (SIGGRAPH Asia ’09). ACM, New York, NY, USA, Article 141, 8 pages. Google ScholarDigital Library
13. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. CoRR abs/1502.01852 (2015). arXiv:1502.01852 http://arxiv.org/abs/1502.01852Google Scholar
14. Binh-Son Hua, Adrien Gruson, Derek Nowrouzezahrai, and Toshiya Hachisuka. 2017. Gradient-Domain Photon Density Estimation. Comput. Graph. Forum 36, 2 (May 2017), 31–38. Google ScholarDigital Library
15. Gao Huang, Zhuang Liu, and Kilian Q. Weinberger. 2016. Densely Connected Convolutional Networks. CoRR abs/1608.06993 (2016). arXiv:1608.06993 http://arxiv.org/abs/1608.06993Google Scholar
16. Forrest N. Iandola, Matthew W. Moskewicz, Khalid Ashraf, Song Han, William J. Dally, and Kurt Keutzer. 2016. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <1MB model size. CoRR abs/1602.07360 (2016). arXiv:1602.07360 http://arxiv.org/abs/1602.07360Google Scholar
17. Wenzel Jakob. 2010. Mitsuba renderer. http://www.mitsuba-renderer.org.Google Scholar
18. Wojciech Jarosz, Derek Nowrouzezahrai, Robert Thomas, Peter-Pike Sloan, and Matthias Zwicker. 2011. Progressive Photon Beams. In Proceedings of the 2011 SIGGRAPH Asia Conference (SA ’11). ACM, New York, NY, USA, Article 181, 12 pages. Google ScholarDigital Library
19. Wojciech Jarosz, Matthias Zwicker, and Henrik Wann Jensen. 2008. The Beam Radiance Estimate for Volumetric Photon Mapping. Comput. Graph. Forum 27 (04 2008), 557–566.Google Scholar
20. Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016. Perceptual losses for real-time style transfer and super-resolution. In European Conference on Computer Vision. Springer, 694–711.Google ScholarCross Ref
21. James T. Kajiya. 1986. The Rendering Equation. SIGGRAPH Comput. Graph. 20, 4 (Aug. 1986), 143–150. Google ScholarDigital Library
22. Nima Khademi Kalantari, Steve Bako, and Pradeep Sen. 2015. A Machine Learning Approach for Filtering Monte Carlo Noise. ACM Transactions on Graphics (TOG) (Proceedings of SIGGRAPH 2015) 34, 4 (2015). Google ScholarDigital Library
23. Markus Kettunen, Erik Härkönen, and Jaakko Lehtinen. 2019. Robust Perceptual Image Similarity via Self-Ensembled CNNs. Manuscript in preparation.Google Scholar
24. Markus Kettunen, Marco Manzi, Miika Aittala, Jaakko Lehtinen, Frédo Durand, and Matthias Zwicker. 2015. Gradient-domain Path Tracing. ACM Trans. Graph. 34, 4, Article 123 (July 2015), 13 pages. Google ScholarDigital Library
25. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. CoRR abs/1412.6980 (2014). arXiv:1412.6980 http://arxiv.org/abs/1412.6980Google Scholar
26. Jaakko Lehtinen, Tero Karras, Samuli Laine, Miika Aittala, Frédo Durand, and Timo Aila. 2013. Gradient-domain Metropolis Light Transport. ACM Trans. Graph. 32, 4, Article 95 (July 2013), 12 pages. Google ScholarDigital Library
27. Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, and Timo Aila. 2018. Noise2Noise: Learning Image Restoration without Clean Data. In Proceedings of the 35th International Conference on Machine Learning, PMLR, Vol. 80.Google Scholar
28. Xiaomeng Li, Hao Chen, Xiaojuan Qi, Qi Dou, Chi-Wing Fu, and Pheng-Ann Heng. 2017. H-DenseUNet: Hybrid Densely Connected UNet for Liver and Liver Tumor Segmentation from CT Volumes. CoRR abs/1709.07330 (2017). arXiv:1709.07330 http://arxiv.org/abs/1709.07330Google Scholar
29. Andrew L. Maas. 2013. Rectifier Nonlinearities Improve Neural Network Acoustic Models.Google Scholar
30. Marco Manzi, Markus Kettunen, Miika Aittala, Jaakko Lehtinen, Frédo Durand, and Matthias Zwicker. 2015. Gradient-Domain Bidirectional Path Tracing. In Proc. Eurographics Symposium on Rendering.Google Scholar
31. Marco Manzi, Markus Kettunen, Frédo Durand, Matthias Zwicker, and Jaakko Lehtinen. 2016a. Temporal Gradient-domain Path Tracing. ACM Trans. Graph. 35, 6, Article 246 (Nov. 2016), 9 pages. Google ScholarDigital Library
32. Marco Manzi, Delio Vicini, and Matthias Zwicker. 2016b. Regularizing Image Reconstruction for Gradient-Domain Rendering with Feature Patches. In Computer graphics forum, Vol. 35. Wiley Online Library, 263–273.Google Scholar
33. Xiao-Jiao Mao, Chunhua Shen, and Yu-Bin Yang. 2016. Image Restoration Using Convolutional Auto-encoders with Symmetric Skip Connections. In Proc. NIPS. Google ScholarDigital Library
34. Michael D. McCool. 1999. Anisotropic Diffusion for Monte Carlo Noise Reduction. ACM Trans. Graph. 18, 2 (April 1999), 171–194. Google ScholarDigital Library
35. Bochang Moon, Steven McDonagh, Kenny Mitchell, and Markus Gross. 2016. Adaptive Polynomial Rendering. ACM Trans. Graph. 35, 4, Article 40 (July 2016), 10 pages. Google ScholarDigital Library
36. Keiron O’Shea and Ryan Nash. 2015. An Introduction to Convolutional Neural Networks. CoRR abs/1511.08458 (2015). arXiv:1511.08458 http://arxiv.org/abs/1511.08458Google Scholar
37. Erik Reinhard, Michael Stark, Peter Shirley, and James Ferwerda. 2002. Photographic Tone Reproduction for Digital Images. ACM Trans. Graph. 21, 3 (July 2002), 267–276. Google ScholarDigital Library
38. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-Net: Convolutional Networks for Biomedical Image Segmentation. CoRR abs/1505.04597 (2015). arXiv:1505.04597 http://arxiv.org/abs/1505.04597Google Scholar
39. Fabrice Rousselle, Wojciech Jarosz, and Jan Novák. 2016. Image-space Control Variates for Rendering. ACM Trans. Graph. 35, 6, Article 169 (Nov. 2016), 12 pages. Google ScholarDigital Library
40. Fabrice Rousselle, Claude Knaus, and Matthias Zwicker. 2011. Adaptive Sampling and Reconstruction Using Greedy Error Minimization. ACM Trans. Graph. 30, 6, Article 159 (Dec. 2011), 12 pages. Google ScholarDigital Library
41. Fabrice Rousselle, Claude Knaus, and Matthias Zwicker. 2012. Adaptive Rendering with Non-local Means Filtering. ACM Trans. Graph. 31, 6, Article 195 (Nov. 2012), 11 pages. Google ScholarDigital Library
42. Fabrice Rousselle, Marco Manzi, and Matthias Zwicker. 2013. Robust Denoising using Feature and Color Information. Computer Graphics Forum 32, 7 (2013), 121–130.Google ScholarCross Ref
43. Tim Salimans and Diederik P. Kingma. 2016. Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks. CoRR abs/1602.07868 (2016). arXiv:1602.07868 http://arxiv.org/abs/1602.07868Google Scholar
44. Karen Simonyan and Andrew Zisserman. 2014. Very Deep Convolutional Networks for Large-Scale Image Recognition. CoRR abs/1409.1556 (2014). arXiv:1409.1556 http://arxiv.org/abs/1409.1556Google Scholar
45. Weilun Sun, Xin Sun, Nathan A. Carr, Derek Nowrouzezahrai, and Ravi Ramamoorthi. 2017. Gradient-Domain Vertex Connection and Merging. In Eurographics Symposium on Rendering – Experimental Ideas & Implementations, Matthias Zwicker and Pedro Sander (Eds.). The Eurographics Association. Google ScholarDigital Library
46. Eric Veach and Leonidas J. Guibas. 1995. Optimally Combining Sampling Techniques for Monte Carlo Rendering. In Proceedings of the 22Nd Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’95). ACM, New York, NY, USA, 419–428. Google ScholarDigital Library
47. Thijs Vogels, Fabrice Rousselle, Brian McWilliams, Gerhard Röthlin, Alex Harvill, David Adler, Mark Meyer, and Jan Novák. 2018. Denoising with Kernel Prediction and Asymmetric Loss Functions. ACM Transactions on Graphics (Proceedings of SIGGRAPH 2018) 37, 4, Article 124 (2018), 124:1–124:15 pages. Google ScholarDigital Library
48. Siming Yan, Feng Shi, Yuhua Chen, Damini Dey, Sang-Eun Lee, Hyuk-Jae Chang, Debiao Li, and Yibin Xie. 2018. Calcium Removal From Cardiac CT Images Using Deep Convolutional Neural Network. CoRR abs/1803.00399 (2018). arXiv:1803.00399 http://arxiv.org/abs/1803.00399Google Scholar
49. Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. 2018. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. CoRR abs/1801.03924 (2018). arXiv:1801.03924 http://arxiv.org/abs/1801.03924Google Scholar
50. Matthias Zwicker, Wojciech Jarosz, Jaakko Lehtinen, Bochang Moon, Ravi Ramamoorthi, Fabrice Rousselle, Pradeep Sen, Cyril Soler, and Sungeui E. Yoon. 2015. Recent Advances in Adaptive Sampling and Reconstruction for Monte Carlo Rendering. Computer Graphics Forum (2015). Google ScholarDigital Library
