“Toward wave-based sound synthesis for computer animation” by Wang, Qu, Langlois and James

  • ©Jui-Hsien Wang, Ante Qu, Timothy R. Langlois, and Doug L. James



Entry Number: 109


    Toward wave-based sound synthesis for computer animation


Session Title: Sounds Good!



    We explore an integrated approach to sound generation that supports a wide variety of physics-based simulation models and computer-animated phenomena. Targeting high-quality offline sound synthesis, we seek to resolve animation-driven sound radiation with near-field scattering and diffraction effects. The core of our approach is a sharp-interface finite-difference time-domain (FDTD) wavesolver, with a series of supporting algorithms to handle rapidly deforming and vibrating embedded interfaces arising in physics-based animation sound. Once the solver rasterizes these interfaces, it must evaluate acceleration boundary conditions (BCs) that involve model-and phenomena-specific computations. We introduce acoustic shaders as a mechanism to abstract away these complexities, and describe a variety of implementations for computer animation: near-rigid objects with ringing and acceleration noise, deformable (finite element) models such as thin shells, bubble-based water, and virtual characters. Since time-domain wave synthesis is expensive, we only simulate pressure waves in a small region about each sound source, then estimate a far-field pressure signal. To further improve scalability beyond multi-threading, we propose a fully time-parallel sound synthesis method that is demonstrated on commodity cloud computing resources. In addition to presenting results for multiple animation phenomena (water, rigid, shells, kinematic deformers, etc.) we also propose 3D automatic dialogue replacement (3DADR) for virtual characters so that pre-recorded dialogue can include character movement, and near-field shadowing and scattering sound effects.


    1. T. Akenine-Möller. 2002. Fast 3D Triangle-box Overlap Testing. J. Graph. Tools 6, 1 (2002). Google ScholarDigital Library
    2. A. Allen and N. Raghuvanshi. 2015. Aerophones in Flatland: Interactive Wave Simulation of Wind Instruments. ACM Transactions on Graphics (Proceedings of SIGGRAPH 2015) 34, 4 (Aug. 2015). Google ScholarDigital Library
    3. S. S. An, D. L. James, and S. Marschner. 2012. Motion-driven Concatenative Synthesis of Cloth Sounds. ACM Transactions on Graphics (SIGGRAPH 2012) (Aug. 2012). Google ScholarDigital Library
    4. Avid Technology. 2018. Pro Tools. (2018). http://www.avid.com/pro-tools.Google Scholar
    5. D. R. Begault. 1994. 3-D Sound for Virtual Reality and Multimedia. Academic Press Professional, Cambridge, MA. Google ScholarDigital Library
    6. S. Bilbao. 2009. Numerical Sound Synthesis: Finite Difference Schemes and Simulation in Musical Acoustics. John Wiley and Sons. Google ScholarDigital Library
    7. S. Bilbao. 2011. Time domain simulation and sound synthesis for the snare drum. J. Acoust. Soc. Am. 131, 1 (2011).Google Scholar
    8. Stefan Bilbao. 2013. Modeling of complex geometries and boundary conditions in finite different/finite volume time domain room acoustics simulation. IEEE Transactions on Audio, Speech, and Language Processing 21 (2013). Google ScholarDigital Library
    9. S. Bilbao and C. J. Webb. 2013. Physical modeling of timpani drums in 3D on GPGPUs. Journal of the Audio Engineering Society 61, 10 (2013), 737–748.Google Scholar
    10. N. Bonneel, G. Drettakis, N. Tsingos, I. Viaud-Delmon, and D. James. 2008. Fast Modal Sounds with Scalable Frequency-Domain Synthesis. ACM Transactions on Graphics 27, 3 (Aug. 2008), 24:1–24:9. Google ScholarDigital Library
    11. D. Botteldooren. 1994. Acoustical finite-difference time-domain simulation in a quasi-cartesian grid. Journal of the Acoustical Society of America 95 (1994).Google Scholar
    12. D. Botteldooren. 1997. Time-domain simulation of the influence of close barriers on sound propagation to the environment. The Journal of the Acoustical Society of America 101, 3 (1997), 1278–1285.Google ScholarCross Ref
    13. J. N. Chadwick, S. S. An, and D. L. James. 2009. Harmonic Shells: A Practical Nonlinear Sound Model for Near-Rigid Thin Shells. ACM Transactions on Graphics (Aug. 2009). Google ScholarDigital Library
    14. J. N. Chadwick and D. L. James. 2011. Animating Fire with Sound. ACM Transactions on Graphics 30, 4 (Aug. 2011). Google ScholarDigital Library
    15. J. N. Chadwick, C. Zheng, and D. L. James. 2012a. Faster Acceleration Noise for Multi-body Animations using Precomputed Soundbanks. ACM Eurographics Symposium on Computer Animation (2012). Google ScholarDigital Library
    16. J. N. Chadwick, C. Zheng, and D. L. James. 2012b. Precomputed Acceleration Noise for Improved Rigid-Body Sound. ACM Transactions on Graphics (Proceedings of SIGGRAPH 2012) 31, 4 (Aug. 2012). Google ScholarDigital Library
    17. A. Chaigne, C. Touzé, and O. Thomas. 2005. Nonlinear vibrations and chaos in gongs and cymbals. Acoustical science and technology 26, 5 (2005), 403–409.Google Scholar
    18. A. Chandak, C. Lauterbach, M. Taylor, Z. Ren, and D. Manocha. 2008. Ad-frustum: Adaptive frustum tracing for interactive sound propagation. IEEE Transactions on Visualization and Computer Graphics 14, 6 (2008), 1707–1722. Google ScholarDigital Library
    19. G. Cirio, D. Li, E. Grinspun, Mi. A. Otaduy, and C. Zheng. 2016. Crumpling sound synthesis. ACM Transactions on Graphics (TOG) 35, 6 (2016), 181. Google ScholarDigital Library
    20. R. Clayton and B. Engquist. 1977. Absorbing boundary conditions for acoustic and elastic wave equations. Bulletin of the Seismological Society of America 67, 6 (1977), 1529.Google ScholarCross Ref
    21. M. Cook. 2015. Pixar, ‘The Road to Point Reyes’ and the long history of landscape in new visual technologies. (2015).Google Scholar
    22. P. R. Cook. 2002. Sound Production and Modeling. IEEE Computer Graphics & Applications 22, 4 (July/Aug. 2002), 23–27. Google ScholarDigital Library
    23. R. L. Cook, L. Carpenter, and E. Catmull. 1987. The Reyes Image Rendering Architecture. In Proceedings of the 14th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’87). ACM, New York, NY, USA, 95–102. Google ScholarDigital Library
    24. M. Ducceschi and C. Touzé. 2015. Modal approach for nonlinear vibrations of damped impacted plates: Application to sound synthesis of gongs and cymbals. Journal of Sound and Vibration 344 (2015), 313–331.Google ScholarCross Ref
    25. B. Engquist and A. Majda. 1977. Absorbing boundary conditions for numerical simulation of waves. Proceedings of the National Academy of Sciences 74, 5 (1977), 1765–1766.Google ScholarCross Ref
    26. Ronald P Fedkiw, Tariq Aslam, Barry Merriman, and Stanley Osher. 1999. A Non-oscillatory Eulerian Approach to Interfaces in Multimaterial Flows (the Ghost Fluid Method). J. Comput. Phys. 152, 2 (1999), 457 — 492. Google ScholarDigital Library
    27. T. Funkhouser, I. Carlbom, G. Elko, G. Pingali, M. Sondhi, and J. West. 1998. A Beam Tracing Approach to Acoustic Modeling for Interactive Virtual Environments. In Proceedings of SIGGRAPH 98 (Computer Graphics Proceedings, Annual Conference Series). 21–32. Google ScholarDigital Library
    28. T. A. Funkhouser, P. Min, and I. Carlbom. 1999. Real-Time Acoustic Modeling for Distributed Virtual Environments. In Proceedings of SIGGRAPH 99 (Computer Graphics Proceedings, Annual Conference Series). 365–374. Google ScholarDigital Library
    29. W. W. Gaver. 1993. Synthesizing auditory icons. In Proceedings of the TNTERACT’93 and CHI’93 conference on Human factors in computing systems. ACM, 228–235. Google ScholarDigital Library
    30. Y. I. Gingold, A. Secord, J. Y. Han, E. Grinspun, and D. Zorin. 2004. A Discrete Model for Inelastic Deformation of Thin Shells.Google Scholar
    31. G. Guennebaud, B.Jacob, et al. 2010. Eigen v3. http://eigen.tuxfamily.org. (2010).Google Scholar
    32. Jon Häggblad and Björn Engquist. 2012. Consistent modeling of boundaries in acoustic finite-difference Time-domain simulations. Journal of the Acoustical Society of America 132 (2012).Google Scholar
    33. P. S. Heckbert. 1987. Ray tracing Jell-O brand gelatin. In ACM SIGGRAPH Computer Graphics, Vol. 21. ACM, 73–74. Google ScholarDigital Library
    34. A. Jacobson, D. Panozzo, et al. 2017. libigl: A simple C++ geometry processing library. (2017). http://libigl.github.io/libigl/.Google Scholar
    35. D. L. James, J. Barbie, and D. K. Pai. 2006. Precomputed Acoustic Transfer: Output-sensitive, accurate sound generation for geometrically complex vibration sources. ACM Transactions on Graphics 25, 3 (July 2006), 987–995. Google ScholarDigital Library
    36. D. L. James and D. K. Pai. 2002. DyRT: Dynamic Response Textures for Real Time Deformation Simulation with Graphics Hardware. ACM Trans. Graph. 21, 3 (July 2002), 582–585. Google ScholarDigital Library
    37. M. Kleiner, B.-I. Dalenbäck, and P. Svensson. 1993. Auralization-An Overview. J. Audio Engineering Society 41 (1993), 861–861. Issue 11.Google Scholar
    38. D. Komatitsch, G. Erlebacher, D. Göddeke, and D. Michéa. 2010. High-order finite-element seismic wave propagation modeling with MPI on a large GPU cluster. Journal of computational physics 229, 20 (2010), 7692–7714. Google ScholarDigital Library
    39. T. R. Langlois, S. S. An, K. K. Jin, and D. L. James. 2014. Eigenmode Compression for Modal Sound Models. ACM Trans. Graph. 33, 4, Article 40 (July 2014), 9 pages. Google ScholarDigital Library
    40. T. R Langlois and D. L James. 2014. Inverse-foley animation: Synchronizing rigid-body motions to sound. ACM Transactions on Graphics (TOG) 33, 4 (2014), 41. Google ScholarDigital Library
    41. T. R. Langlois, C. Zheng, and D. L. James. 2016. Toward Animating Water with Complex Acoustic Bubbles. ACM Trans. Graph. 35, 4, Article 95 (July 2016), 13 pages. Google ScholarDigital Library
    42. S. Larsson and V Thomée. 2009. Partial Differential Equations with Numerical Methods. Springer.Google Scholar
    43. Q.-H. Liu and J. Tao. 1997. The perfectly matched layer for acoustic waves in absorptive media. The Journal of the Acoustical Society of America 102, 4 (1997), 2072–2082.Google ScholarCross Ref
    44. S. Marburg and B. Nolte. 2008. Computational acoustics of noise propagation in fluids: finite and boundary element methods. Vol. 578. Springer.Google Scholar
    45. R. Mehra, N. Raghuvanshi, L. Antani, A. Chandak, S. Curtis, and D. Manocha. 2013. Wave-based sound propagation in large open scenes using an equivalent source formulation. ACM Transactions on Graphics (TOG) 32, 2 (2013), 19. Google ScholarDigital Library
    46. R. Mehra, N. Raghuvanshi, L. Savioja, M. C. Lin, and D. Manocha. 2012. An efficient GPU-based time domain solver for the acoustic wave equation. Applied Acoustics 73, 2(2012), 83–94.Google ScholarCross Ref
    47. A. Meshram, R. Mehra, H. Yang, E. Dunn, J.-M. Frahm, and D. Manochak. 2014. P-hrtf: Efficient personalized hrtf computation for high-fidelity spatial sound. Mixed and Augmented Reality (ISMAR), 2014 IEEE International Symposium on (2014).Google ScholarCross Ref
    48. P. Micikevicius. 2009. 3D Finite Difference Computation on GPUs Using CUDA. In Proceedings of 2Nd Workshop on General Purpose Processing on Graphics Processing Units (GPGPU-2). ACM, New York, NY, USA, 79–84. Google ScholarDigital Library
    49. M. Minnaert. 1933. XVI. On musical air-bubbles and the sounds of running water. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science 16, 104 (1933), 235–248.Google ScholarCross Ref
    50. R. Mittal, H. Dong, M. Bozkurttas, F. M. Najjar, A. Vargas, and A. Loebbecke. 2008. A versatile sharp interface immersed boundary method for incompressible flows with complex boundaries. J. Comput. Phys. 227 (2008). Google ScholarDigital Library
    51. R. Mittal and G. Iaccarino. 2005. Immersed Boundary Methods. Annual Review of Fluid Mechanics 37 (2005).Google Scholar
    52. P. Morse and K. U. Ingard. 1968. Theoretical Acoustics. Princeton University Press, Princeton, New Jersey.Google Scholar
    53. W. Moss, H. Yeh, J.-M. Hong, M. C. Lin, and D. Manocha. 2010. Sounding Liquids: Automatic Sound Synthesis from Fluid Simulation. ACM Trans. Graph. 29, 3 (2010). Google ScholarDigital Library
    54. J. F. O’Brien, P. R. Cook, and G. Essl. 2001. Synthesizing Sounds From Physically Based Motion. In Proceedings of SIGGRAPH 2001. 529–536. Google ScholarDigital Library
    55. J. F. O’Brien, C. Shen, and C. M. Gatchalian. 2002. Synthesizing sounds from rigid-body simulations. In The ACM SIGGRAPH 2002 Symposium on Computer Animation. ACM Press, 175–181. Google ScholarDigital Library
    56. C. S. Peskin. 1981. The fluid dynamics of heart valves: experimental, theoretical and computational methods. Annual Review of Fluid Mechanics 14 (1981).Google Scholar
    57. N. Raghuvanshi, R. Narain, and M. C. Lin. 2009. Efficient and Accurate Sound Propagation Using Adaptive Rectangular Decomposition. IEEE Trans. Vis. Comput. Graph. 15, 5 (2009), 789–801. Google ScholarDigital Library
    58. N. Raghuvanshi and J. Snyder. 2014. Parametric wave field coding for precomputed sound propagation. ACM Transactions on Graphics (TOG) 33, 4 (2014), 38. Google ScholarDigital Library
    59. J. Saarelma, J. Botts, B. Hamilton, and L. Savioja. 2016. Audibility of dispersion error in room acoustic finite-difference time-domain simulation as a function of simulation distance. The Journal of the Acoustical Society of America 139, 4 (2016), 1822–1832.Google ScholarCross Ref
    60. C. Schissler, R. Mehra, and D. Manocha. 2014. High-order diffraction and diffuse reflections for interactive sound propagation in large environments. ACM Transactions on Graphics (TOG) 33, 4 (2014), 39. Google ScholarDigital Library
    61. C. Schreck, D. Rohmer, D. James, S. Hahmann, and M.-P. Cani. 2016. Real-time sound synthesis for paper material based on geometric analysis. In Eurographics/ACM SIGGRAPH Symposium on Computer Animation (2016). Google ScholarDigital Library
    62. E. Schweickart, D. L. James, and S. Marschner. 2017. Animating Elastic Rods with Sound. ACM Transactions on Graphics 36, 4 (July 2017). Google ScholarDigital Library
    63. A. A. Shabana. 2012. Theory of Vibration: An Introduction. Springer Science & Business Media.Google Scholar
    64. A. A. Shabana. 2013. Dynamics of multibody systems. Cambridge university press.Google Scholar
    65. Side Effects. 2018. Houdini Engine. (2018). http://www.sidefx.com.Google Scholar
    66. J. O. Smith. 1992. Physical modeling using digital waveguides. Computer music journal 16, 4 (1992), 74–91.Google Scholar
    67. A. Taflove and S. C. Hagness. 2005. Computational Electrodynamics: The Finite-Difference Time-Domain Method. Artech House.Google Scholar
    68. T. Takala and J. Hahn. 1992. Sound rendering. In Computer Graphics (Proceedings of SIGGRAPH 92). 211–220. Google ScholarDigital Library
    69. J. G. Tolan and J. B. Schneider. 2003. Locally conformal method for acoustic finite-difference time-domain modeling of rigid surfaces. Journal of the Acoustical Society of America 114 (2003).Google Scholar
    70. N. Tsingos, T. Funkhouser, A. Ngan, and I. Carlbom. 2001. Modeling acoustics in virtual environments using the uniform theory of diffraction. In SIGGRAPH ’01: Proceedings of the 28th annual conference on Computer graphics and interactive techniques. ACM, New York, NY, USA, 545–552. Google ScholarDigital Library
    71. K. van den Doel. 2005. Physically Based Models for Liquid Sounds. ACM Trans. Appl. Percept. 2, 4 (Oct. 2005), 534–546. Google ScholarDigital Library
    72. K. van den Doel, P. G. Kry, and D. K. Pai. 2001. FoleyAutomatic: Physically-based Sound Effects for Interactive Simulation and Animation. (2001), 537–544. Google ScholarDigital Library
    73. K. van den Doel and D. K. Pai. 1998. The sounds of physical shapes. Presence: Teleoperators and Virtual Environments 7, 4 (1998), 382–395. Google ScholarDigital Library
    74. M. Vorländer. 2008. Auralization. Aachen: Springer (2008).Google Scholar
    75. C.J. Webb. 2014. Parallel computation techniques for virtual acoustics and physical modelling synthesis. Ph.D. Dissertation.Google Scholar
    76. C. J. Webb and S. Bilbao. 2011. Computing room acoustics with CUDA – 3D FDTD schemes with boundary losses and viscosity. 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (2011), 317–320.Google ScholarCross Ref
    77. H. Yeh, R. Mehra, Z. Ren, L. Antani, D. Manocha, and M. Lin. 2013. Wave-ray Coupling for Interactive Sound Propagation in Large Complex Scenes. ACM Trans. Graph. 32, 6, Article 165 (Nov. 2013), 11 pages. Google ScholarDigital Library
    78. C. Zheng and D. L. James. 2009. Harmonic Fluids. ACM Transactions on Graphics (SIGGRAPH 2009) 28, 3 (Aug. 2009). Google ScholarDigital Library
    79. C. Zheng and D. L. James. 2010. Rigid-Body Fracture Sound with Precomputed Sound-banks. ACM Transactions on Graphics (SIGGRAPH 2010) 29, 3 (July 2010). Google ScholarDigital Library
    80. C. Zheng and D. L. James. 2011. Toward High-Quality Modal Contact Sound. ACM Transactions on Graphics (Proceedings of SIGGRAPH 2011) 30, 4 (Aug. 2011). Google ScholarDigital Library

ACM Digital Library Publication: