“Deep scattering: rendering atmospheric clouds with radiance-predicting neural networks” by Kallweit, Müller, Mcwilliams, Gross and Novák
Conference:
Type(s):
Title:
- Deep scattering: rendering atmospheric clouds with radiance-predicting neural networks
Session/Category Title: Rendering and Sampling
Presenter(s)/Author(s):
Abstract:
We present a technique for efficiently synthesizing images of atmospheric clouds using a combination of Monte Carlo integration and neural networks. The intricacies of Lorenz-Mie scattering and the high albedo of cloud-forming aerosols make rendering of clouds—e.g. the characteristic silverlining and the “whiteness” of the inner body—challenging for methods based solely on Monte Carlo integration or diffusion theory. We approach the problem differently. Instead of simulating all light transport during rendering, we pre-learn the spatial and directional distribution of radiant flux from tens of cloud exemplars. To render a new scene, we sample visible points of the cloud and, for each, extract a hierarchical 3D descriptor of the cloud geometry with respect to the shading location and the light source. The descriptor is input to a deep neural network that predicts the radiance function for each shading configuration. We make the key observation that progressively feeding the hierarchical descriptor into the network enhances the network’s ability to learn faster and predict with higher accuracy while using fewer coefficients. We also employ a block design with residual connections to further improve performance. A GPU implementation of our method synthesizes images of clouds that are nearly indistinguishable from the reference solution within seconds to minutes. Our method thus represents a viable solution for applications such as cloud design and, thanks to its temporal stability, for high-quality production of animated content.
References:
1. Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, et al. 2015. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. (2015). http://tensorflow.org/ Software available from tensorflow.org.
2. Steve Bako, Thijs Vogels, Brian McWilliams, Mark Meyer, Jan Novák, Alex Harvill, Pradeep Sen, Tony DeRose, and Fabrice Rousselle. 2017. Kernel-Predicting Convolutional Networks for Denoising Monte Carlo Renderings. ACM TOG (Proc. of SIGGRAPH) 36, 4 (2017).
3. David Balduzzi, Marcus Frean, Lennox Leary, JP Lewis, Kurt Wan-Duo Ma, and Brian McWilliams. 2017. The Shattered Gradients Problem: If resnets are the answer, then what is the question?. In Proceedings of The 34th International Conference on Machine Learning.
4. Yoshua Bengio, Aaron Courville, and Pascal Vincent. 2013. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence 35, 8 (2013), 1798–1828.
5. Benedikt Bitterli and Wojciech Jarosz. 2017. Beyond Points and Beams: Higher-Dimensional Photon Samples for Volumetric Light Transport. ACM Transactions on Graphics (Proceedings of SIGGRAPH) 36, 4 (July 2017).
6. Antoine Bouthors, Fabrice Neyret, Nelson Max, Eric Bruneton, and Cyril Crassin. 2008. Interactive multiple anisotropic scattering in clouds. In Proc. of Symposium on Interactive 3D Graphics and Games.
7. Chakravarty R. A. Chaitanya, Anton Kaplanyan, Christoph Schied, Marco Salvi, Aaron Lefohn, Derek Nowrouzezahrai, and Timo Aila. 2017. Interactive Reconstruction of Noisy Monte Carlo Image Sequences using a Recurrent Autoencoder. ACM Trans. Graph. (Proc. SIGGRAPH) (2017).
8. Subrahmanyan Chandrasekhar. 1960. Radiative transfer. Dover Publications.
9. Mengyu Chu and Nils Thuerey. 2017. Data-Driven Synthesis of Smoke Flows with CNN-based Feature Descriptors. ACM TOG (Proc. of SIGGRAPH) 36, 4 (2017).
10. Eugene d’Eon and Geoffrey Irving. 2011. A quantized-diffusion model for rendering translucent materials. ACM TOG (Proc. of SIGGRAPH) 30, 4 (July 2011), 56:1–56:14.
11. Yoshinori Dobashi, Kazufumi Kaneda, Hideo Yamashita, Tsuyoshi Okita, and Tomoyuki Nishita. 2000. A Simple, Efficient Method for Realistic Animation of Clouds. In Proc. of SIGGRAPH 00 (Annual Conference Series). ACM Press/Addison-Wesley Publishing Co., New York, NY, USA, 19–28.
12. Craig Donner and Henrik Wann Jensen. 2005. Light diffusion in multi-layered translucent materials. ACM TOG (Proc. of SIGGRAPH) 24, 3 (July 2005), 1032–1039.
13. Oskar Elek, Tobias Ritschel, Carsten Dachsbacher, and Hans-Peter Seidel. 2014. Principal-Ordinates Propagation for real-time rendering of participating media. Computers & Graphics 45 (2014), 28–39.
14. Oskar Elek, Tobias Ritschel, Alexander Wilkie, and Hans-Peter Seidel. 2012. Interactive cloud rendering using temporally coherent photon mapping. Computers & Graphics 36, 8 (2012), 1109–1118.
15. Jeppe Revall Frisvad, Niels Jørgen Christensen, and Henrik Wann Jensen. 2007. Computing the scattering properties of participating media using lorenz-mie theory. ACM TOG (Proc. of SIGGRAPH) 26, 3, Article 60 (jul 2007).
16. Jeppe Revall Frisvad, Toshiya Hachisuka, and Thomas Kim Kjeldsen. 2014. Directional Dipole Model for Subsurface Scattering. ACM TOG 34, 1 (Dec. 2014), 5:1–5:12.
17. Geoffrey Y. Gardner. 1985. Visual Simulation of Clouds. Computer Graphics (Proc. of SIGGRAPH) 19, 3 (July 1985), 297–304.
18. Iliyan Georgiev, Jaroslav Křivánek, Toshiya Hachisuka, Derek Nowrouzezahrai, and Wojciech Jarosz. 2013. Joint importance sampling of low-order volumetric scattering. ACM TOG (Proc. of SIGGRAPH Asia) 32, 6 (Nov. 2013), 164:1–164:14.
19. Mark J. Harris and Anselmo Lastra. 2001. Real-Time Cloud Rendering. Computer Graphics Forum 20, 3 (2001), 76–85. Cross Ref
20. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
21. Wojciech Jarosz, Craig Donner, Matthias Zwicker, and Henrik Wann Jensen. 2008. Radiance caching for participating media. ACM TOG 27, 1 (March 2008), 7:1–7:11.
22. Wojciech Jarosz, Derek Nowrouzezahrai, Iman Sadeghi, and Henrik Wann Jensen. 2011. A comprehensive theory of volumetric radiance estimation using photon points and beams. ACM TOG 30, 1 (Feb. 2011), 5:1–5:19.
23. Wojciech Jarosz, Matthias Zwicker, and Henrik Wann Jensen. 2008. The beam radiance estimate for volumetric photon mapping. Computer Graphics Forum (Proc. of Eurographics) 27, 2 (April 2008), 557–566.
24. Henrik Wann Jensen and Per H. Christensen. 1998. Efficient simulation of light transport in scenes with participating media using photon maps. In Proc. of SIGGRAPH 98 (Annual Conference Series). ACM, New York, NY, USA, 311–320.
25. Henrik Wann Jensen, Stephen R. Marschner, Marc Levoy, and Pat Hanrahan. 2001. A practical model for subsurface light transport. In Proc. of SIGGRAPH 01 (Annual Conference Series). ACM, New York, NY, USA, 511–518.
26. James T. Kajiya and Brian P Von Herzen. 1984. Ray tracing volume densities. Computer Graphics (Proc. of SIGGRAPH) 18, 3 (Jan. 1984), 165–174.
27. Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
28. David Koerner, Jamie Portsmouth, Filip Sadlo, Thomas Ertl, and Bernd Eberhardt. 2014. Flux-Limited Diffusion for Multiple Scattering in Participating Media. Computer Graphics Forum 33, 6 (Sept. 2014), 178–189.
29. Christopher Kulla and Marcos Fajardo. 2012. Importance sampling techniques for path tracing in participating media. CGF (Proc. of Eurographics Symposium on Rendering) 31, 4 (June 2012), 1519–1528.
30. Peter Kutz, Ralf Habel, Yining Karl Li, and Jan Novák. 2017. Spectral and Decomposition Tracking for Rendering Heterogeneous Volumes. ACM TOG (Proc. of SIGGRAPH) 36, 4 (2017).
31. Jaroslav Křivánek, Iliyan Georgiev, Toshiya Hachisuka, Petr Vevoda, Martin Šik, Derek Nowrouzezahrai, and Wojciech Jarosz. 2014. Unifying points, beams, and paths in volumetric light transport simulation. ACM TOG (Proc. of SIGGRAPH) 33, 4 (2014), 103:1–103:13.
32. Eric P. Lafortune and Yves D. Willems. 1996. Rendering participating media with bidirectional path tracing. In Proc. of Eurographics Workshop on Rendering Techniques. Springer-Verlag, London, UK, 91–100.
33. Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep Learning. Nature 521 (2015), 436–444. Cross Ref
34. Richard Lee and Carol O’Sullivan. 2007. Accelerated Light Propagation Through Participating Media. In Proc. Eurographics / IEEE VGTC Conference on Volume Graphics. 17–23.
35. Ludwig Lorenz. 1890. Lysbevægelsen i og uden for en af plane Lysbølger belyst Kugle. In Det Kongelige Danske Videnskabernes Selskabs Skrifter (trykt utg.): Naturvidenskabelig og Mathematisk Afdeling.
36. Gustav Mie. 1908. Beiträge zur Optik trüber Medien, speziell kolloidaler Metallösungen. Annalen der Physik 330 (1908), 377–445. Cross Ref
37. Jonathan T. Moon, Bruce Walter, and Stephen R. Marschner. 2007. Rendering Discrete Random Media Using Precomputed Scattering Solutions. In Proc. of EGSR. Eurographics Association, 231–242.
38. Thomas Müller, Marios Papas, Markus Gross, Wojciech Jarosz, and Jan Novák. 2016. Efficient Rendering of Heterogenous Polydisperse Granular Media. ACM TOG (Proc. of SIGGRAPH Asia) 35, 6 (2016), 168:1–168:14.
39. Oliver Nalbach, Elena Arabadzhiyska, Dushyant Mehta, Hans-Peter Seidel, and Tobias Ritschel. 2017. Deep Shading: Convolutional Neural Networks for Screen-Space Shading. CGF (Proc. of Eurographics Symposium on Rendering) 36, 4 (2017).
40. Tomoyuki Nishita, Yoshinori Dobashi, and Eihachiro Nakamae. 1996. Display of Clouds Taking into Account Multiple Anisotropic Scattering and Sky Light. In Proc. of SIGGRAPH 96. ACM, New York, NY, USA, 379–386.
41. Jan Novák, Derek Nowrouzezahrai, Carsten Dachsbacher, and Wojciech Jarosz. 2012. Virtual ray lights for rendering scenes with participating media. ACM TOG (Proc. of SIGGRAPH) 31, 4 (July 2012), 60:1–60:11.
42. Jan Novák, Andrew Selle, and Wojciech Jarosz. 2014. Residual ratio tracking for estimating attenuation in participating media. ACM TOG (Proc. of SIGGRAPH Asia) 33, 6 (Nov. 2014), 179:1–179:11.
43. Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. 2016a. Wavenet: A Generative Model for Raw Audio. arXiv preprint arXiv:1609.03499 (2016).
44. Aaron Van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. 2016b. Pixel Recurrent Neural Networks. In Proceedings of The 33rd International Conference on Machine Learning. 1747–1756.
45. Mark Pauly, Thomas Kollig, and Alexander Keller. 2000. Metropolis light transport for participating media. In Proc. of Eurographics Workshop on Rendering Techniques. Springer-Verlag, London, UK, 11–22.
46. Matthias Raab, Daniel Seibert, and Alexander Keller. 2008. Unbiased global illumination with participating media. In Monte Carlo and Quasi-Monte Carlo Methods 2006. Springer, 591–606.
47. Peiran Ren, Jiaping Wang, Minmin Gong, Stephen Lin, Xin Tong, and Baining Guo. 2013. Global Illumination with Radiance Regression Functions. ACM TOG (Proc. of SIGGRAPH) 32, 4 (July 2013), 130:1–130:12.
48. Kirk Riley, David S. Ebert, Martin Kraus, Jerry Tessendorf, and Charles Hansen. 2004. Efficient Rendering of Atmospheric Phenomena. In Proc. of EGSR. Eurographics Association, Aire-la-Ville, Switzerland, 375–386.
49. Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. 2016. Progressive Neural Networks. CoRR abs/1606.04671 (2016). http://arxiv.org/abs/1606.04671
50. Karen Simonyan and Andrew Zisserman. 2014. Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv preprint arXiv:1409.1556 (2014).
51. Jos Stam. 1995. Multiple scattering as a diffusion process. In Proc. of Eurographics Workshop on Rendering Techniques. Springer-Verlag, 41–50. Cross Ref
52. László Szirmay-Kalos, Iliyan Georgiev, Milán Magdics, Balázs Molnár, and Dávid Legrády. 2017. Unbiased Estimators to Render Procedurally Generated Inhomogeneous Participating Media. Computer Graphics Forum (Proc. of Eurographics) 36, 2 (2017).
53. László Szirmay-Kalos, Mateu Sbert, and Tamás Ummenhoffer. 2005. Real-time Multiple Scattering in Participating Media with Illumination Networks. In Proc. of EGSR. Eurographics Association, Aire-la-Ville, Switzerland, 277–282.
54. László Szirmay-Kalos, Balázs Tóth, and Milán Magdics. 2011. Free path sampling in high resolution inhomogeneous participating media. Computer Graphics Forum 30, 1 (2011), 85–97. Cross Ref
55. Helmut K. Weickmann and Hans J. aufm Kampe. 1953. Physical properties of cumulus clouds. Journal of Meteorology 10, 3 (1953), 204–211. <0204:PPOCC>2.0.CO;2 Cross Ref
56. E. R. Woodcock, T. Murphy, P. J. Hemmings, and T. C. Longworth. 1965. Techniques used in the GEM code for Monte Carlo neutronics calculations in reactors and other systems of complex geometry. In Applications of Computing Methods to Reactor Problems. Argonne National Laboratory.
57. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation. arXiv preprint arXiv:1609.08144 (2016).
58. Yonghao Yue, Kei Iwasaki, Bing-Yu Chen, Yoshinori Dobashi, and Tomoyuki Nishita. 2011. Toward optimal space partitioning for unbiased, adaptive free path sampling of inhomogeneous participating media. CGF (Proc. of Pacific Graphics) 30, 7 (2011), 1911–1919. Cross Ref
59. Yonghao Yue, Kei Iwasaki, Bing-Yu Chen, Yoshinori Dobashi, and Tomoyuki Nishita. 2010. Unbiased, adaptive stochastic sampling for rendering inhomogeneous participating media. ACM TOG (Proc. of SIGGRAPH Asia) 29, 6 (Dec. 2010), 177:1–177:8.
