“GradNet: unsupervised deep screened poisson reconstruction for gradient-domain rendering” by Guo, Li, Li, Qiang, Hu, et al. …
Conference:
Type(s):
Title:
- GradNet: unsupervised deep screened poisson reconstruction for gradient-domain rendering
Session/Category Title: Samples & Speckles
Presenter(s)/Author(s):
Moderator(s):
Abstract:
Monte Carlo (MC) methods for light transport simulation are flexible and general but typically suffer from high variance and slow convergence. Gradientdomain rendering alleviates this problem by additionally generating image gradients and reformulating rendering as a screened Poisson image reconstruction problem. To improve the quality and performance of the reconstruction, we propose a novel and practical deep learning based approach in this paper. The core of our approach is a multi-branch auto-encoder, termed GradNet, which end-to-end learns a mapping from a noisy input image and its corresponding image gradients to a high-quality image with low variance. Once trained, our network is fast to evaluate and does not require manual parameter tweaking. Due to the difficulty in preparing ground-truth images for training, we design and train our network in a completely unsupervised manner by learning directly from the input data. This is the first solution incorporating unsupervised deep learning into the gradient-domain rendering framework. The loss function is defined as an energy function including a data fidelity term and a gradient fidelity term. To further reduce the noise of the reconstructed image, the loss function is reinforced by adding a regularizer constructed from selected rendering-specific features. We demonstrate that our method improves the reconstruction quality for a diverse set of scenes, and reconstructing a high-resolution image takes far less than one second on a recent GPU.
References:
1. Jonghee Back, Sung-Eui Yoon, and Bochang Moon. 2018. Feature Generation for Adaptive Gradient-Domain Path Tracing. Computer Graphics Forum 37, 7 (2018), 65–74.Google ScholarCross Ref
2. Steve Bako, Thijs Vogels, Brian Mcwilliams, Mark Meyer, Jan NováK, Alex Harvill, Pradeep Sen, Tony Derose, and Fabrice Rousselle. 2017. Kernel-predicting Convolutional Networks for Denoising Monte Carlo Renderings. ACM Trans. Graph. 36, 4 (July 2017), 97:1–97:14.Google ScholarDigital Library
3. Pablo Bauszat, Martin Eisemann, and Marcus Magnor. 2011. Guided Image Filtering for Interactive High-quality Global Illumination. Computer Graphics Forum (2011).Google Scholar
4. Pablo Bauszat, Victor Petitjean, and Elmar Eisemann. 2017. Gradient-domain Path Reusing. ACM Trans. Graph. 36, 6 (Nov. 2017), 229:1–229:9.Google ScholarDigital Library
5. Pravin Bhat, Brian Curless, Michael Cohen, and C. Lawrence Zitnick. 2008. Fourier Analysis of the 2D Screened Poisson Equation for Gradient Domain Problems. In Computer Vision – ECCV 2008. Springer Berlin Heidelberg, 114–128.Google ScholarDigital Library
6. Pravin Bhat, C. Lawrence Zitnick, Michael Cohen, and Brian Curless. 2010. GradientShop: A Gradient-domain Optimization Framework for Image and Video Filtering. ACM Trans. Graph. 29, 2 (April 2010), 10:1–10:14.Google ScholarDigital Library
7. Benedikt Bitterli. 2016. Rendering resources. https://benedikt-bitterli.me/resources/.Google Scholar
8. Benedikt Bitterli, Fabrice Rousselle, Bochang Moon, Jose A. Iglesias-Guitian, David Adler, Kenny Mitchell, Wojciech Jarosz, and Jan Novak. 2016. Nonlinearly Weighted First-order Regression for Denoising Monte Carlo Renderings. Computer Graphics Forum (2016).Google Scholar
9. Malik Boughida and Tamy Boubekeur. 2017. Bayesian Collaborative Denoising for Monte Carlo Rendering. Computer Graphics Forum (Proc. EGSR 2017) 36, 4 (2017), 137–153.Google Scholar
10. A. Buades, B. Coll, and J. Morel. 2005. A Review of Image Denoising Algorithms, with a New One. Multiscale Modeling & Simulation 4, 2 (2005), 490–530.Google ScholarCross Ref
11. Chakravarty R. Alla Chaitanya, Anton S. Kaplanyan, Christoph Schied, Marco Salvi, Aaron Lefohn, Derek Nowrouzezahrai, and Timo Aila. 2017. Interactive Reconstruction of Monte Carlo Image Sequences Using a Recurrent Denoising Autoencoder. ACM Trans. Graph. 36, 4 (July 2017), 98:1–98:12.Google Scholar
12. Mauricio Delbracio, Pablo Musé, Antoni Buades, Julien Chauvier, Nicholas Phelps, and Jean-Michel Morel. 2014. Boosting Monte Carlo Rendering by Ray Histogram Fusion. ACM Trans. Graph. 33, 1 (Feb. 2014), 8:1–8:15. Google ScholarDigital Library
13. Qingnan Fan, Jiaolong Yang, David Wipf, Baoquan Chen, and Xin Tong. 2018. Image Smoothing via Unsupervised Learning. ACM Trans. Graph. 37, 6 (Dec. 2018), 259:1–259:14.Google ScholarDigital Library
14. Ian J. Goodfellow, Jean Pouget-abadie, Mehdi Mirza, Bing Xu, David Warde-farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in neural information processing systems (NIPS).Google Scholar
15. Adrien Gruson, Binh-Son Hua, Nicolas Vibert, Derek Nowrouzezahrai, and Toshiya Hachisuka. 2018. Gradient-domain Volumetric Photon Density Estimation. ACM Transactions on Graphics (2018).Google Scholar
16. Saerom Ha, Sojin Oh, Jonghee Back, Sung-Eui Yoon, and Bochang Moon. 2019. Gradient Outlier Removal for Gradient-Domain Path Tracing. In Eurographics 2019.Google Scholar
17. K. He, X. Zhang, S. Ren, and J. Sun. 2015. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. In 2015 IEEE International Conference on Computer Vision (ICCV). 1026–1034.Google Scholar
18. K. He, X. Zhang, S. Ren, and J. Sun. 2016. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 770–778.Google Scholar
19. Pedro Hermosilla, Sebastian Maisch, Tobias Ritschel, and Timo Ropinski. 2018. Deep-learning the Latent Space of Light Transport. CoRR abs/1811.04756 (2018). arXiv:1811.04756 http://arxiv.org/abs/1811.04756Google Scholar
20. G. E. Hinton and R. R. Salakhutdinov. 2006. Reducing the Dimensionality of Data with Neural Networks. Science 313, 5786 (2006), 504–507. Google ScholarCross Ref
21. Binh-Son Hua, Adrien Gruson, Derek Nowrouzezahrai, and Toshiya Hachisuka. 2017. Gradient-Domain Photon Density Estimation. Eurographics (2017).Google Scholar
22. Binh-Son Hua, Adrien Gruson, Victor Petitjean, Matthias Zwicker, Derek Nowrouzezahrai, Elmar Eisemann, and Toshiya Hachisuka. 2019. A Survey on Gradient-Domain Rendering. Computer Graphics Forum (Proceedings of Eurographics – State of the Art Reports) 38, 2 (2019).Google Scholar
23. Wenzel Jakob. 2010. Mitsuba renderer. http://www.mitsuba-renderer.org.Google Scholar
24. Nima Khademi Kalantari, Steve Bako, and Pradeep Sen. 2015. A Machine Learning Approach for Filtering Monte Carlo Noise. ACM Trans. Graph. 34, 4 (July 2015), 122:1–122:12.Google ScholarDigital Library
25. Nima Khademi Kalantari and Ravi Ramamoorthi. 2017. Deep High Dynamic Range Imaging of Dynamic Scenes. ACM Transactions on Graphics (Proceedings of SIGGRAPH 2017) 36, 4 (2017).Google Scholar
26. Nima Khademi Kalantari and Pradeep Sen. 2013. Removing the Noise in Monte Carlo Rendering with General Image Denoising Algorithms. Computer Graphics Forum 32, 2pt1 (2013), 93–102.Google Scholar
27. Simon Kallweit, Thomas Müller, Brian Mcwilliams, Markus Gross, and Jan Novák. 2017. Deep Scattering: Rendering Atmospheric Clouds with Radiance-predicting Neural Networks. ACM Trans. Graph. 36, 6 (Nov. 2017), 231:1–231:11.Google ScholarDigital Library
28. Kaizhang Kang, Zimin Chen, Jiaping Wang, Kun Zhou, and Hongzhi Wu. 2018. Efficient Reflectance Capture Using an Autoencoder. ACM Trans. Graph. 37, 4 (July 2018), 127:1–127:10.Google ScholarDigital Library
29. A. Keller, L. Fascione, M. Fajardo, I. Georgiev, P. Christensen, J. Hanika, C. Eisenacher, and G. Nichols. 2015. The Path Tracing Revolution in the Movie Industry. In ACM SIGGRAPH 2015 Courses (SIGGRAPH ’15). 24:1–24:7.Google Scholar
30. Alexander Keller, Jaroslav Křivánek, Jan Novák, Anton Kaplanyan, and Marco Salvi. 2018. Machine Learning and Rendering. In ACM SIGGRAPH 2018 Courses. 19:1–19:2.Google Scholar
31. Markus Kettunen, Erik Härkönen, and Jaakko Lehtinen. 2019. Deep Convolutional Reconstruction for Gradient-domain Rendering. ACM Trans. Graph. 38, 4 (July 2019), 126:1–126:12.Google ScholarDigital Library
32. Markus Kettunen, Marco Manzi, Miika Aittala, Jaakko Lehtinen, Frédo Durand, and Matthias Zwicker. 2015. Gradient-domain Path Tracing. ACM Trans. Graph. 34, 4 (July 2015), 123:1–123:13.Google ScholarDigital Library
33. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. CoRR abs/1412.6980 (2015).Google Scholar
34. Alexandr Kuznetsov, Nima Khademi Kalantari, and Ravi Ramamoorthi. 2018. Deep Adaptive Sampling for Low Sample Count Rendering. Computer Graphics Forum 37 (2018), 35–44.Google ScholarCross Ref
35. Jaroslav Křivánek, Christophe Chevallier, Vladimir Koylazov, Ondřej Karlíik, Henrik Wann Jensen, and Thomas Ludwig. 2018. Realistic Rendering in Architecture and Product Visualization. In ACM SIGGRAPH 2018 Courses (SIGGRAPH ’18). Article 10, 5 pages.Google Scholar
36. Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. Nature 521 (2015), 436. Google ScholarCross Ref
37. Jaakko Lehtinen, Tero Karras, Samuli Laine, Miika Aittala, Frédo Durand, and Timo Aila. 2013. Gradient-domain Metropolis Light Transport. ACM Trans. Graph. 32, 4 (July 2013), 95:1–95:12.Google ScholarDigital Library
38. Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, and Timo Aila. 2018. Noise2Noise: Learning Image Restoration without Clean Data. In Proceedings of the 35th International Conference on Machine Learning (Proceedings of Machine Learning Research), Vol. 80. PMLR, 2965–2974.Google Scholar
39. Tzu-Mao Li, Yu-Ting Wu, and Yung-Yu Chuang. 2012. SURE-based Optimization for Adaptive Sampling and Reconstruction. ACM Transactions on Graphics (Proceedings of ACM SIGGRAPH Asia 2012) 31, 6 (November 2012), 186:1–186:9.Google Scholar
40. Andrew L. Maas, Awni Y. Hannun, and Andrew Y. Ng. 2013. Rectifier nonlinearities improve neural network acoustic models. In ICML Workshop on Deep Learning for Audio, Speech and Language Processing.Google Scholar
41. Marco Manzi, Markus Kettunen, Miika Aittala, Jaakko Lehtinen, Frédo Durand, and Matthias Zwicker. 2015. Gradient-Domain Bidirectional Path Tracing. In Eurographics Symposium on Rendering, Jaakko Lehtinen and Derek Nowrouzezahrai (Eds.). The Eurographics Association.Google Scholar
42. Marco Manzi, Markus Kettunen, Frédo Durand, Matthias Zwicker, and Jaakko Lehtinen. 2016a. Temporal Gradient-domain Path Tracing. ACM Trans. Graph. 35, 6 (Nov. 2016), 246:1–246:9.Google ScholarDigital Library
43. Marco Manzi, Fabrice Rousselle, Markus Kettunen, Jaakko Lehtinen, and Matthias Zwicker. 2014. Improved Sampling for Gradient-domain Metropolis Light Transport. ACM Trans. Graph. 33, 6 (Nov. 2014), 178:1–178:12.Google ScholarDigital Library
44. Marco Manzi, Delio Vicini, and Matthias Zwicker. 2016b. Regularizing Image Reconstruction for Gradient-Domain Rendering with Feature Patches. Computer Graphics Forum 35, 2 (2016), 263–273.Google ScholarCross Ref
45. Bochang Moon, Nathan Carr, and Sung-Eui Yoon. 2014. Adaptive Rendering Based on Weighted Local Regression. ACM Trans. Graph. 33, 5 (Sept. 2014), 170:1–170:14.Google ScholarDigital Library
46. Bochang Moon, Steven McDonagh, Kenny Mitchell, and Markus Gross. 2016. Adaptive Polynomial Rendering. ACM Trans. Graph. 35, 4 (July 2016), 40:1–40:10.Google ScholarDigital Library
47. Thomas Müller, Brian McWilliams, Fabrice Rousselle, Markus Gross, and Jan Novák. 2018. Neural Importance Sampling. CoRR abs/1808.03856 (2018). arXiv:1808.03856 http://arxiv.org/abs/1808.03856Google Scholar
48. Oliver Nalbach, Elena Arabadzhiyska, Dushyant Mehta, Hans-Peter Seidel, and Tobias Ritschel. 2017. Deep Shading: Convolutional Neural Networks for Screen-Space Shading. 36, 4 (2017).Google Scholar
49. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch.Google Scholar
50. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-Net: Convolutional Networks for Biomedical Image Segmentation. In Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015. 234–241.Google ScholarCross Ref
51. Fabrice Rousselle, Wojciech Jarosz, and Jan Novák. 2016. Image-space Control Variates for Rendering. ACM Trans. Graph. 35, 6 (Nov. 2016), 169:1–169:12.Google ScholarDigital Library
52. Fabrice Rousselle, Claude Knaus, and Matthias Zwicker. 2012. Adaptive Rendering with Non-local Means Filtering. ACM Trans. Graph. 31, 6 (Nov. 2012), 195:1–195:11.Google ScholarDigital Library
53. Fabrice Rousselle, Marco Manzi, and Matthias Zwicker. 2013. Robust Denoising using Feature and Color Information. Computer Graphics Forum 32, 7 (2013), 121–130.Google ScholarCross Ref
54. Pradeep Sen and Soheil Darabi. 2012. On Filtering the Noise from the Random Parameters in Monte Carlo Rendering. ACM Trans. Graph. 31, 3 (May 2012), 18:1–18:15.Google ScholarDigital Library
55. Pradeep Sen, Matthias Zwicker, Fabrice Rousselle, Sung-Eui Yoon, and Nima Khademi Kalantari. 2015. Denoising Your Monte Carlo Renders: Recent Advances in Image-space Adaptive Sampling and Reconstruction. In ACM SIGGRAPH 2015 Courses (SIGGRAPH ’15). 11:1–11:255.Google ScholarDigital Library
56. Weilun Sun, Xin Sun, Nathan A. Carr, Derek Nowrouzezahrai, and Ravi Ramamoorthi. 2017. Gradient-Domain Vertex Connection and Merging. In Eurographics Symposium on Rendering. The Eurographics Association.Google Scholar
57. Thijs Vogels, Fabrice Rousselle, Brian Mcwilliams, Gerhard Röthlin, Alex Harvill, David Adler, Mark Meyer, and Jan Novák. 2018. Denoising with Kernel Prediction and Asymmetric Loss Functions. ACM Trans. Graph. 37, 4 (July 2018), 124:1–124:15.Google ScholarDigital Library
58. H. Zhao, O. Gallo, I. Frosio, and J. Kautz. 2017. Loss Functions for Image Restoration With Neural Networks. IEEE Transactions on Computational Imaging 3, 1 (2017), 47–57.Google ScholarCross Ref
59. Quan Zheng and Matthias Zwicker. 2019. Learning to importance sample in primary sample space. In Computer Graphics Forum, Vol. 38. Wiley Online Library.Google Scholar
60. Matthias Zwicker, Wojciech Jarosz, Jaakko Lehtinen, Bochang Moon, Ravi Ramamoorthi, Fabrice Rousselle, Pradeep Sen, Cyril Soler, and Sung-Eui Yoon. 2015. Recent Advances in Adaptive Sampling and Reconstruction for Monte Carlo Rendering. Computer Graphics Forum (Proceedings of Eurographics – State of the Art Reports) 34, 2 (may 2015), 667–681.Google ScholarCross Ref
