“FoleyAutomatic: physically-based sound effects for interactive simulation and animation” by van den Doel, Kry and Pai

We describe algorithms for real-time synthesis of realistic sound effects for interactive simulations (e.g., games) and animation. These sound effects are produced automatically, from 3D models using dynamic simulation and user interaction. We develop algorithms that are efficient, physically-based, and can be controlled by users in natural ways. We develop effective techniques for producing high quality continuous contact sounds from dynamic simulations running at video rates which are slow relative to audio synthesis. We accomplish this using modal models driven by contact forces modeled at audio rates, which are much higher than the graphics frame rate. The contact forces can be computed from simulations or can be custom designed. We demonstrate the effectiveness with complex realistic simulations.

1. T. Ananthapadmanaban and V. Radhakrishnan. An investigation on the role of surface irregularities in the noise spectrum of rolling and sliding contacts. Wear, 83:399-409, 1982.]]
2. D. R. Begault. 3-D Sound for Virtual Reality and Multimedia. Academic Press, London, 1994.]]
3. W. Buxton. Introduction to this special issue on nonspeech audio. Human Computer Interaction, 4(1):1-9, 1989.]]
4. W. Buxton. Using our ears: an introduction to the use of nonspeech audio cues. In E. Farrell, editor, Extracting meaning from complex data: processing, display, interaction. Proceedings of the SPIE, volume Vol 1259, pages 124-127, 1990.]]
5. A. Chaigne and V. Doutaut. Numerical simulations of xylophones. I. Time domain modeling of the vibrating bars. J. Acoust. Soc. Am., 101(1):539-557, 1997.]]
6. A. Chaudhary, A. Freed, S. Khoury, and D. Wessel. A 3- D Graphical User Interface for Resonance Modeling. In Proceedings of the International Computer Music Conference, Ann Arbor, 1998.]]
7. P. R. Cook. Integration of physical modeling for synthesis and animation. In Proceedings of the International Computer Music Conference, pages 525-528, Banff, 1995.]]
8. P. R. Cook. Physically informed sonic modeling (PhISM): Percussive synthesis. In Proceedings of the International Computer Music Conference, pages 228- 231, Hong Kong, 1996.]]
9. P. R. Cook and G. Scavone. The synthesis toolkit (STK), version 2.1. In Proc. of the International Computer Music Conference, Beijing, 1999.]]
10. D. DiFilippo and D. K. Pai. The AHI: An audio and haptic interface for contact interactions. In UIST’00 (13th Annual ACM Symposium on User Interface Software and Technology), 2000.]]
11. A. Freed and X. Depalle. Synthesis of Hundreds of Sinusoidal Partials on a Desktop Computer without Custom Hardware. In Proceedings of The International Conference on Signal Processing Applications and Technology, Santa Clara, 1993.]]
12. D. J. Fried. Auditory correlates of perceived mallet hardness for a set of recorded percussive sound events. J. Acoust. Soc. Am., 87(1):311-321, 1990.]]
13. T. A. Funkhouser, P. Min, and I. Carlbom. Real-time acoustic modeling for distributed virtual environments. Proc. SIGGRAPH 99, ACM Computer Graphics, 1999.]]
14. W. W. Gaver. Everyday listening and auditory icons. PhD thesis, University of California in San Diego, 1988.]]
15. W. W. Gaver. Synthesizing auditory icons. In Proceedings of the ACM INTERCHI 1993, pages 228-235, 1993.]]
16. W. W. Gaver. What in the world do we hear?: An ecological approach to auditory event perception. Ecological Psychology, 5(1):1-29, 1993.]]
17. J. K. Hahn, H. Fouad, L. Gritz, and J. W. Lee. Integrating sounds and motions in virtual environments. In Sound for Animation and Virtual Reality, SIGGRAPH 95 Course 10 Notes, 1995.]]
18. D. J. Hermes. Synthesis of the sounds produced by rolling balls. In Internal IPO report no. 1226, IPO, Center for User-System Interaction, Eindhoven, The Netherlands, 2000.]]
19. D. P. Hess and A. Soom. Normal vibrations and friction under harmonic loads. II. Rough planar contacts. Transactions of the ASME. Journal of Tribology, 113:87-92, 1991.]]
20. M. M. J. Houben, D. J. Hermes, and A. Kohlrausch. Auditory perception of the size and velocity of rolling balls. In IPO Annual Progress Report, volume 34, 1999.]]
21. D. Jaffe. Ten criteria for evaluating synthesis and processing techniques. Computer Music Journal, 19(1):76- 87, 1995.]]
22. K. L. Johnson. Contact Mechanics. Cambridge University Press, Cambridge, 1985.]]
23. R. L. Klatzky, D. K. Pai, and E. P. Krotkov. Hearing material: Perception of material from contact sounds. PRESENCE: Teleoperators and Virtual Environments, 9(4):399-410, 2000.]]
24. M. V. Mathews. The Technology of Computer Music. MIT Press, Cambridge, 1969.]]
25. J. D. Morrison and J.-M. Adrien. Mosaic: A framework for modal synthesis. Computer Music Journal, 17(1), 1993.]]
26. P. R. Nayak. Contact vibrations. Journal of Sound and Vibration, 22:297-322, 1972.]]
27. D. D. Nelson, D. E. Johnson, and E. Cohen. Haptic rendering of surface-to-surface sculpted model interaction. In Proceedings of the ASME Dynamic Systems and Control Division, volume DSC-Vol. 67, pages 101-108, 1999.]]
28. J. F. O’Brien, P. R. Cook, and G. Essl. Synthesizing Sounds from Physically Based Motion. In SIGGRAPH 01, 2001.]]
29. D. K. Pai, U. M. Ascher, and P. G. Kry. Forward Dynamics Algorithms for Multibody Chains and Contact. In Proceedings of the 2000 IEEE International Conference on Robotics and Automation, pages 857-863, 2000.]]
30. D. K. Pai, K. van den Doel, D. L. James, J. Lang, J. E. Lloyd, J. L. Richmond, and S. H. Yau. Scanning physical interaction behavior of 3D objects. In Computer Graphics (ACM SIGGRAPH 2001 Conference Proceedings), 2001.]]
31. X. Rodet. Musical Sound Signal Analysis/Synthesis: Sinusoidal + Residual and Elementary Waveform Models. In IEEE Time-Frequency and Time-Scale Workshop 97, Coventry, Grande Bretagne, 1997.]]
32. L. Savioja, J. Huopaniemi, T. Lokki, and R. Vnnen. Virtual environment simulation – Advances in the DIVA project. In Proc. Int. Conf. Auditory Display, Palo Alto, USA, 1997.]]
33. R. S. Sayles and T. R. Thomas. Surface topography as a non-stationary random process. Nature, 271:431-434, 1978.]]
34. X. Serra. A System for Sound Analysis / Transformation / Synthesis Based on a Deterministic Plus Stochastic Decomposition. PhD thesis, Dept. of Music, Stanford University, 1989.]]
35. J. O. Smith. Physical modeling using digital waveguides. Computer Music Journal, 16(4):75-87, 1992.]]
36. J. Stam. Evaluation of loop subdivision surfaces. In SIGGRAPH 98, 1998. Included on course notes CD-ROM.]]
37. K. Steiglitz. A Digital Signal Processing Primer with Applications to Digital Audio and Computer Music. Addison-Wesley, New York, 1996.]]
38. D. Stoianovici and Y. Hurmuzlu. A critical study of the applicability of rigid-body collision theory. ASME Journal of Applied Mechanics, 63:307-316, 1996.]]
39. T. Takala and J. Hahn. Sound rendering. Proc. SIG- GRAPH 92, ACM Computer Graphics, 26(2):211-220, 1992.]]
40. T. R. Thomas. Rough Surfaces. Imperial College Press, London, second edition, 1999.]]
41. D. J. Thompson, D. F. Fodiman, and H. Mahe. Experimental validation of the twins prediction program for rolling noise, parts I and II. Journal of Sound and Vibration, 193:123-147, 1996.]]
42. N. Tsingos, T. Funkhouser, A. Ngan, and I. Carlbom. Modeling acoustics in virtual environments using the uniform theory of diffraction. In SIGGRAPH 01, 2001.]]
43. K. van den Doel. Sound Synthesis for Virtual Reality and Computer Games. PhD thesis, University of British Columbia, 1998.]]
44. K. van den Doel and D. K. Pai. Synthesis of shape dependent sounds with physical modeling. In Proceedings of the International Conference on Auditory Displays 1996, Palo Alto, 1996.]]
45. K. van den Doel and D. K. Pai. The sounds of physical shapes. Presence, 7(4):382-395, 1998.]]
46. J. Wawrzynek. VLSI models for real-time music synthesis. In M. Mathews and J. Pierce, editors, Current Directions in Computer Music Research. MIT Press, 1989.]]