“Toward wave-based sound synthesis for computer animation” by Wang, Qu, Langlois and James
Conference:
Type:
Entry Number: 109
Title:
- Toward wave-based sound synthesis for computer animation
Session/Category Title: Sounds Good!
Presenter(s)/Author(s):
Moderator(s):
Abstract:
We explore an integrated approach to sound generation that supports a wide variety of physics-based simulation models and computer-animated phenomena. Targeting high-quality offline sound synthesis, we seek to resolve animation-driven sound radiation with near-field scattering and diffraction effects. The core of our approach is a sharp-interface finite-difference time-domain (FDTD) wavesolver, with a series of supporting algorithms to handle rapidly deforming and vibrating embedded interfaces arising in physics-based animation sound. Once the solver rasterizes these interfaces, it must evaluate acceleration boundary conditions (BCs) that involve model-and phenomena-specific computations. We introduce acoustic shaders as a mechanism to abstract away these complexities, and describe a variety of implementations for computer animation: near-rigid objects with ringing and acceleration noise, deformable (finite element) models such as thin shells, bubble-based water, and virtual characters. Since time-domain wave synthesis is expensive, we only simulate pressure waves in a small region about each sound source, then estimate a far-field pressure signal. To further improve scalability beyond multi-threading, we propose a fully time-parallel sound synthesis method that is demonstrated on commodity cloud computing resources. In addition to presenting results for multiple animation phenomena (water, rigid, shells, kinematic deformers, etc.) we also propose 3D automatic dialogue replacement (3DADR) for virtual characters so that pre-recorded dialogue can include character movement, and near-field shadowing and scattering sound effects.
References:
1. T. Akenine-Möller. 2002. Fast 3D Triangle-box Overlap Testing. J. Graph. Tools 6, 1 (2002). Google ScholarDigital Library
2. A. Allen and N. Raghuvanshi. 2015. Aerophones in Flatland: Interactive Wave Simulation of Wind Instruments. ACM Transactions on Graphics (Proceedings of SIGGRAPH 2015) 34, 4 (Aug. 2015). Google ScholarDigital Library
3. S. S. An, D. L. James, and S. Marschner. 2012. Motion-driven Concatenative Synthesis of Cloth Sounds. ACM Transactions on Graphics (SIGGRAPH 2012) (Aug. 2012). Google ScholarDigital Library
4. Avid Technology. 2018. Pro Tools. (2018). http://www.avid.com/pro-tools.Google Scholar
5. D. R. Begault. 1994. 3-D Sound for Virtual Reality and Multimedia. Academic Press Professional, Cambridge, MA. Google ScholarDigital Library
6. S. Bilbao. 2009. Numerical Sound Synthesis: Finite Difference Schemes and Simulation in Musical Acoustics. John Wiley and Sons. Google ScholarDigital Library
7. S. Bilbao. 2011. Time domain simulation and sound synthesis for the snare drum. J. Acoust. Soc. Am. 131, 1 (2011).Google Scholar
8. Stefan Bilbao. 2013. Modeling of complex geometries and boundary conditions in finite different/finite volume time domain room acoustics simulation. IEEE Transactions on Audio, Speech, and Language Processing 21 (2013). Google ScholarDigital Library
9. S. Bilbao and C. J. Webb. 2013. Physical modeling of timpani drums in 3D on GPGPUs. Journal of the Audio Engineering Society 61, 10 (2013), 737–748.Google Scholar
10. N. Bonneel, G. Drettakis, N. Tsingos, I. Viaud-Delmon, and D. James. 2008. Fast Modal Sounds with Scalable Frequency-Domain Synthesis. ACM Transactions on Graphics 27, 3 (Aug. 2008), 24:1–24:9. Google ScholarDigital Library
11. D. Botteldooren. 1994. Acoustical finite-difference time-domain simulation in a quasi-cartesian grid. Journal of the Acoustical Society of America 95 (1994).Google Scholar
12. D. Botteldooren. 1997. Time-domain simulation of the influence of close barriers on sound propagation to the environment. The Journal of the Acoustical Society of America 101, 3 (1997), 1278–1285.Google ScholarCross Ref
13. J. N. Chadwick, S. S. An, and D. L. James. 2009. Harmonic Shells: A Practical Nonlinear Sound Model for Near-Rigid Thin Shells. ACM Transactions on Graphics (Aug. 2009). Google ScholarDigital Library
14. J. N. Chadwick and D. L. James. 2011. Animating Fire with Sound. ACM Transactions on Graphics 30, 4 (Aug. 2011). Google ScholarDigital Library
15. J. N. Chadwick, C. Zheng, and D. L. James. 2012a. Faster Acceleration Noise for Multi-body Animations using Precomputed Soundbanks. ACM Eurographics Symposium on Computer Animation (2012). Google ScholarDigital Library
16. J. N. Chadwick, C. Zheng, and D. L. James. 2012b. Precomputed Acceleration Noise for Improved Rigid-Body Sound. ACM Transactions on Graphics (Proceedings of SIGGRAPH 2012) 31, 4 (Aug. 2012). Google ScholarDigital Library
17. A. Chaigne, C. Touzé, and O. Thomas. 2005. Nonlinear vibrations and chaos in gongs and cymbals. Acoustical science and technology 26, 5 (2005), 403–409.Google Scholar
18. A. Chandak, C. Lauterbach, M. Taylor, Z. Ren, and D. Manocha. 2008. Ad-frustum: Adaptive frustum tracing for interactive sound propagation. IEEE Transactions on Visualization and Computer Graphics 14, 6 (2008), 1707–1722. Google ScholarDigital Library
19. G. Cirio, D. Li, E. Grinspun, Mi. A. Otaduy, and C. Zheng. 2016. Crumpling sound synthesis. ACM Transactions on Graphics (TOG) 35, 6 (2016), 181. Google ScholarDigital Library
20. R. Clayton and B. Engquist. 1977. Absorbing boundary conditions for acoustic and elastic wave equations. Bulletin of the Seismological Society of America 67, 6 (1977), 1529.Google ScholarCross Ref
21. M. Cook. 2015. Pixar, ‘The Road to Point Reyes’ and the long history of landscape in new visual technologies. (2015).Google Scholar
22. P. R. Cook. 2002. Sound Production and Modeling. IEEE Computer Graphics & Applications 22, 4 (July/Aug. 2002), 23–27. Google ScholarDigital Library
23. R. L. Cook, L. Carpenter, and E. Catmull. 1987. The Reyes Image Rendering Architecture. In Proceedings of the 14th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’87). ACM, New York, NY, USA, 95–102. Google ScholarDigital Library
24. M. Ducceschi and C. Touzé. 2015. Modal approach for nonlinear vibrations of damped impacted plates: Application to sound synthesis of gongs and cymbals. Journal of Sound and Vibration 344 (2015), 313–331.Google ScholarCross Ref
25. B. Engquist and A. Majda. 1977. Absorbing boundary conditions for numerical simulation of waves. Proceedings of the National Academy of Sciences 74, 5 (1977), 1765–1766.Google ScholarCross Ref
26. Ronald P Fedkiw, Tariq Aslam, Barry Merriman, and Stanley Osher. 1999. A Non-oscillatory Eulerian Approach to Interfaces in Multimaterial Flows (the Ghost Fluid Method). J. Comput. Phys. 152, 2 (1999), 457 — 492. Google ScholarDigital Library
27. T. Funkhouser, I. Carlbom, G. Elko, G. Pingali, M. Sondhi, and J. West. 1998. A Beam Tracing Approach to Acoustic Modeling for Interactive Virtual Environments. In Proceedings of SIGGRAPH 98 (Computer Graphics Proceedings, Annual Conference Series). 21–32. Google ScholarDigital Library
28. T. A. Funkhouser, P. Min, and I. Carlbom. 1999. Real-Time Acoustic Modeling for Distributed Virtual Environments. In Proceedings of SIGGRAPH 99 (Computer Graphics Proceedings, Annual Conference Series). 365–374. Google ScholarDigital Library
29. W. W. Gaver. 1993. Synthesizing auditory icons. In Proceedings of the TNTERACT’93 and CHI’93 conference on Human factors in computing systems. ACM, 228–235. Google ScholarDigital Library
30. Y. I. Gingold, A. Secord, J. Y. Han, E. Grinspun, and D. Zorin. 2004. A Discrete Model for Inelastic Deformation of Thin Shells.Google Scholar
31. G. Guennebaud, B.Jacob, et al. 2010. Eigen v3. http://eigen.tuxfamily.org. (2010).Google Scholar
32. Jon Häggblad and Björn Engquist. 2012. Consistent modeling of boundaries in acoustic finite-difference Time-domain simulations. Journal of the Acoustical Society of America 132 (2012).Google Scholar
33. P. S. Heckbert. 1987. Ray tracing Jell-O brand gelatin. In ACM SIGGRAPH Computer Graphics, Vol. 21. ACM, 73–74. Google ScholarDigital Library
34. A. Jacobson, D. Panozzo, et al. 2017. libigl: A simple C++ geometry processing library. (2017). http://libigl.github.io/libigl/.Google Scholar
35. D. L. James, J. Barbie, and D. K. Pai. 2006. Precomputed Acoustic Transfer: Output-sensitive, accurate sound generation for geometrically complex vibration sources. ACM Transactions on Graphics 25, 3 (July 2006), 987–995. Google ScholarDigital Library
36. D. L. James and D. K. Pai. 2002. DyRT: Dynamic Response Textures for Real Time Deformation Simulation with Graphics Hardware. ACM Trans. Graph. 21, 3 (July 2002), 582–585. Google ScholarDigital Library
37. M. Kleiner, B.-I. Dalenbäck, and P. Svensson. 1993. Auralization-An Overview. J. Audio Engineering Society 41 (1993), 861–861. Issue 11.Google Scholar
38. D. Komatitsch, G. Erlebacher, D. Göddeke, and D. Michéa. 2010. High-order finite-element seismic wave propagation modeling with MPI on a large GPU cluster. Journal of computational physics 229, 20 (2010), 7692–7714. Google ScholarDigital Library
39. T. R. Langlois, S. S. An, K. K. Jin, and D. L. James. 2014. Eigenmode Compression for Modal Sound Models. ACM Trans. Graph. 33, 4, Article 40 (July 2014), 9 pages. Google ScholarDigital Library
40. T. R Langlois and D. L James. 2014. Inverse-foley animation: Synchronizing rigid-body motions to sound. ACM Transactions on Graphics (TOG) 33, 4 (2014), 41. Google ScholarDigital Library
41. T. R. Langlois, C. Zheng, and D. L. James. 2016. Toward Animating Water with Complex Acoustic Bubbles. ACM Trans. Graph. 35, 4, Article 95 (July 2016), 13 pages. Google ScholarDigital Library
42. S. Larsson and V Thomée. 2009. Partial Differential Equations with Numerical Methods. Springer.Google Scholar
43. Q.-H. Liu and J. Tao. 1997. The perfectly matched layer for acoustic waves in absorptive media. The Journal of the Acoustical Society of America 102, 4 (1997), 2072–2082.Google ScholarCross Ref
44. S. Marburg and B. Nolte. 2008. Computational acoustics of noise propagation in fluids: finite and boundary element methods. Vol. 578. Springer.Google Scholar
45. R. Mehra, N. Raghuvanshi, L. Antani, A. Chandak, S. Curtis, and D. Manocha. 2013. Wave-based sound propagation in large open scenes using an equivalent source formulation. ACM Transactions on Graphics (TOG) 32, 2 (2013), 19. Google ScholarDigital Library
46. R. Mehra, N. Raghuvanshi, L. Savioja, M. C. Lin, and D. Manocha. 2012. An efficient GPU-based time domain solver for the acoustic wave equation. Applied Acoustics 73, 2(2012), 83–94.Google ScholarCross Ref
47. A. Meshram, R. Mehra, H. Yang, E. Dunn, J.-M. Frahm, and D. Manochak. 2014. P-hrtf: Efficient personalized hrtf computation for high-fidelity spatial sound. Mixed and Augmented Reality (ISMAR), 2014 IEEE International Symposium on (2014).Google ScholarCross Ref
48. P. Micikevicius. 2009. 3D Finite Difference Computation on GPUs Using CUDA. In Proceedings of 2Nd Workshop on General Purpose Processing on Graphics Processing Units (GPGPU-2). ACM, New York, NY, USA, 79–84. Google ScholarDigital Library
49. M. Minnaert. 1933. XVI. On musical air-bubbles and the sounds of running water. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science 16, 104 (1933), 235–248.Google ScholarCross Ref
50. R. Mittal, H. Dong, M. Bozkurttas, F. M. Najjar, A. Vargas, and A. Loebbecke. 2008. A versatile sharp interface immersed boundary method for incompressible flows with complex boundaries. J. Comput. Phys. 227 (2008). Google ScholarDigital Library
51. R. Mittal and G. Iaccarino. 2005. Immersed Boundary Methods. Annual Review of Fluid Mechanics 37 (2005).Google Scholar
52. P. Morse and K. U. Ingard. 1968. Theoretical Acoustics. Princeton University Press, Princeton, New Jersey.Google Scholar
53. W. Moss, H. Yeh, J.-M. Hong, M. C. Lin, and D. Manocha. 2010. Sounding Liquids: Automatic Sound Synthesis from Fluid Simulation. ACM Trans. Graph. 29, 3 (2010). Google ScholarDigital Library
54. J. F. O’Brien, P. R. Cook, and G. Essl. 2001. Synthesizing Sounds From Physically Based Motion. In Proceedings of SIGGRAPH 2001. 529–536. Google ScholarDigital Library
55. J. F. O’Brien, C. Shen, and C. M. Gatchalian. 2002. Synthesizing sounds from rigid-body simulations. In The ACM SIGGRAPH 2002 Symposium on Computer Animation. ACM Press, 175–181. Google ScholarDigital Library
56. C. S. Peskin. 1981. The fluid dynamics of heart valves: experimental, theoretical and computational methods. Annual Review of Fluid Mechanics 14 (1981).Google Scholar
57. N. Raghuvanshi, R. Narain, and M. C. Lin. 2009. Efficient and Accurate Sound Propagation Using Adaptive Rectangular Decomposition. IEEE Trans. Vis. Comput. Graph. 15, 5 (2009), 789–801. Google ScholarDigital Library
58. N. Raghuvanshi and J. Snyder. 2014. Parametric wave field coding for precomputed sound propagation. ACM Transactions on Graphics (TOG) 33, 4 (2014), 38. Google ScholarDigital Library
59. J. Saarelma, J. Botts, B. Hamilton, and L. Savioja. 2016. Audibility of dispersion error in room acoustic finite-difference time-domain simulation as a function of simulation distance. The Journal of the Acoustical Society of America 139, 4 (2016), 1822–1832.Google ScholarCross Ref
60. C. Schissler, R. Mehra, and D. Manocha. 2014. High-order diffraction and diffuse reflections for interactive sound propagation in large environments. ACM Transactions on Graphics (TOG) 33, 4 (2014), 39. Google ScholarDigital Library
61. C. Schreck, D. Rohmer, D. James, S. Hahmann, and M.-P. Cani. 2016. Real-time sound synthesis for paper material based on geometric analysis. In Eurographics/ACM SIGGRAPH Symposium on Computer Animation (2016). Google ScholarDigital Library
62. E. Schweickart, D. L. James, and S. Marschner. 2017. Animating Elastic Rods with Sound. ACM Transactions on Graphics 36, 4 (July 2017). Google ScholarDigital Library
63. A. A. Shabana. 2012. Theory of Vibration: An Introduction. Springer Science & Business Media.Google Scholar
64. A. A. Shabana. 2013. Dynamics of multibody systems. Cambridge university press.Google Scholar
65. Side Effects. 2018. Houdini Engine. (2018). http://www.sidefx.com.Google Scholar
66. J. O. Smith. 1992. Physical modeling using digital waveguides. Computer music journal 16, 4 (1992), 74–91.Google Scholar
67. A. Taflove and S. C. Hagness. 2005. Computational Electrodynamics: The Finite-Difference Time-Domain Method. Artech House.Google Scholar
68. T. Takala and J. Hahn. 1992. Sound rendering. In Computer Graphics (Proceedings of SIGGRAPH 92). 211–220. Google ScholarDigital Library
69. J. G. Tolan and J. B. Schneider. 2003. Locally conformal method for acoustic finite-difference time-domain modeling of rigid surfaces. Journal of the Acoustical Society of America 114 (2003).Google Scholar
70. N. Tsingos, T. Funkhouser, A. Ngan, and I. Carlbom. 2001. Modeling acoustics in virtual environments using the uniform theory of diffraction. In SIGGRAPH ’01: Proceedings of the 28th annual conference on Computer graphics and interactive techniques. ACM, New York, NY, USA, 545–552. Google ScholarDigital Library
71. K. van den Doel. 2005. Physically Based Models for Liquid Sounds. ACM Trans. Appl. Percept. 2, 4 (Oct. 2005), 534–546. Google ScholarDigital Library
72. K. van den Doel, P. G. Kry, and D. K. Pai. 2001. FoleyAutomatic: Physically-based Sound Effects for Interactive Simulation and Animation. (2001), 537–544. Google ScholarDigital Library
73. K. van den Doel and D. K. Pai. 1998. The sounds of physical shapes. Presence: Teleoperators and Virtual Environments 7, 4 (1998), 382–395. Google ScholarDigital Library
74. M. Vorländer. 2008. Auralization. Aachen: Springer (2008).Google Scholar
75. C.J. Webb. 2014. Parallel computation techniques for virtual acoustics and physical modelling synthesis. Ph.D. Dissertation.Google Scholar
76. C. J. Webb and S. Bilbao. 2011. Computing room acoustics with CUDA – 3D FDTD schemes with boundary losses and viscosity. 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (2011), 317–320.Google ScholarCross Ref
77. H. Yeh, R. Mehra, Z. Ren, L. Antani, D. Manocha, and M. Lin. 2013. Wave-ray Coupling for Interactive Sound Propagation in Large Complex Scenes. ACM Trans. Graph. 32, 6, Article 165 (Nov. 2013), 11 pages. Google ScholarDigital Library
78. C. Zheng and D. L. James. 2009. Harmonic Fluids. ACM Transactions on Graphics (SIGGRAPH 2009) 28, 3 (Aug. 2009). Google ScholarDigital Library
79. C. Zheng and D. L. James. 2010. Rigid-Body Fracture Sound with Precomputed Sound-banks. ACM Transactions on Graphics (SIGGRAPH 2010) 29, 3 (July 2010). Google ScholarDigital Library
80. C. Zheng and D. L. James. 2011. Toward High-Quality Modal Contact Sound. ACM Transactions on Graphics (Proceedings of SIGGRAPH 2011) 30, 4 (Aug. 2011). Google ScholarDigital Library
