“Perceptual audio rendering of complex virtual environments” by Tsingos, Gallo and Drettakis

We propose a real-time 3D audio rendering pipeline for complex virtual scenes containing hundreds of moving sound sources. The approach, based on auditory culling and spatial level-of-detail, can handle more than ten times the number of sources commonly available on consumer 3D audio hardware, with minimal decrease in audio quality. The method performs well for both indoor and outdoor environments. It leverages the limited capabilities of audio hardware for many applications, including interactive architectural acoustics simulations and automatic 3D voice management for video games.Our approach dynamically eliminates inaudible sources and groups the remaining audible sources into a budget number of clusters. Each cluster is represented by one impostor sound source, positioned using perceptual criteria. Spatial audio processing is then performed only on the impostor sound sources rather than on every original source thus greatly reducing the computational cost.A pilot validation study shows that degradation in audio quality, as well as localization impairment, are limited and do not seem to vary significantly with the cluster budget. We conclude that our real-time perceptual audio rendering pipeline can generate spatialized audio for complex auditory environments without introducing disturbing changes in the resulting perceived soundfield.

1. ANSI. 1978. American national standard method for the calculation of the absorption of sound by the atmosphere. ANSI S1.26-1978, American Institute of Physics (for Acoustical Society of America), New York.]]Google Scholar
2. BEGAULT, D. R., MCCLAIN, B. U., AND ANDERSON, M. R. 2001. Early reflection thresholds for virtual sound sources. In Proc. 2001 Int. Workshop on Spatial Media.]]Google Scholar
3. BEGAULT, D. R. 1994. 3D Sound for Virtual Reality and Multimedia. Academic Press Professional.]] Google ScholarDigital Library
4. BLAUERT, J. 1983. Spatial Hearing: The Psychophysics of Human Sound Localization. M.I.T. Press, Cambridge, MA.]]Google Scholar
5. BORISH, J. 1984. Extension of the image model to arbitrary polyhedra. J. of the Acoustical Society of America 75, 6.]]Google ScholarCross Ref
6. BRANDENBURG, K. 1999. mp3 and AAC explained. AES 17th International Conference on High-Quality Audio Coding (Sept.).]]Google Scholar
7. BREGMAN, A. 1990. Auditory Scene Analysis, The perceptual school of sound. The MIT Press.]]Google Scholar
8. CHEN, J., VEEN, B. V., AND HECOX, K. 1995. A spatial feature extraction and regularization model for the head-related transfer function. J. of the Acoustical Society of America 97 (Jan.), 439–452.]]Google ScholarCross Ref
9. CHEN, H., WALLACE, G., GUPTA, A., LI, K., FUNKHOUSER, T., AND COOK, P. 2002. Experiences with scalability of display walls. Proceedings of the Immersive Projection Technology (IPT) Workshop (Mar.).]]Google Scholar
10. DIRECT SOUND 3D, 2004. Direct X homepage, Microsoft©. http://www.microsoft.com/windows/directx/default.asp.]]Google Scholar
11. DOBASHI, Y., YAMAMOTO, T., AND NISHITA, T. 2003. Real-time rendering of aerodynamic sound using sound textures based on computational fluid dynamics. ACM Transactions on Graphics 22, 3 (Aug.), 732–740. (Proceedings of ACM SIGGRAPH 2003).]] Google ScholarDigital Library
12. EAX, 2004. Environmental audio extensions 4.0, Creative©. http://www.soundblaster.com/eaudio.]]Google Scholar
13. ELLIS, D. 1992. A perceptual representation of audio. Master’s thesis, Massachusets Institute of Technology.]]Google Scholar
14. FALLER, C., AND BAUMGARTE, F. 2002. Binaural cue coding applied to audio compression with flexible rendering. In Proc. 113th AES Convention.]]Google Scholar
15. FILIPANITS, F. 1994. Design and implementation of an auralization system with a spectrum-based temporal processing optimization. Master thesis, Univ. of Miami.]]Google Scholar
16. FOUAD, H., HAHN, J., AND BALLAS, J. 1997. Perceptually based scheduling algorithms for real-time synthesis of complex sonic environments. proceedings of the 1997 International Conference on Auditory Display (ICAD’97), Xerox Palo Alto Research Center, Palo Alto, USA.]]Google Scholar
17. FOUAD, H., BALLAS, J., AND BROCK, D. 2000. An extensible toolkit for creating virtual sonic environments. Proceedings of Intl. Conf. on Auditory Display (Atlanta, USA, May 2000).]]Google Scholar
18. FUNKHOUSER, T., AND SEQUIN, C. 1993. Adaptive display algorithms for interactive frame rates during visualization of complex virtual environments. Computer Graphics (SIGGRAPH ’93 proceedings), Los Angeles, CA (August), 247–254.]] Google ScholarDigital Library
19. FUNKHOUSER, T., MIN, P., AND CARLBOM, I. 1999. Real-time acoustic modeling for distributed virtual environments. ACM Computer Graphics, SIGGRAPH ’99 Proceedings (Aug.), 365–374.]] Google ScholarDigital Library
20. GARDNER, W. 1997. Reverberation algorithms. In Applications of Digital Signal Processing to Audio and Acoustics, M. Kahrs and K. Brandenburg, Eds. Kluwer Academic Publishers, 85–131.]]Google Scholar
21. GREWIN, C. 1993. Methods for quality assessment of low bit-rate audio codecs. proceedings of the 12th AES conference, 97–107.]]Google Scholar
22. HERDER, J. 1999. Optimization of sound spatialization resource management through clustering. The Journal of Three Dimensional Images, 3D-Forum Society 13, 3 (Sept.), 59–65.]]Google Scholar
23. HERDER, J. 1999. Visualization of a clustering algorithm of sound sources based on localization errors. The Journal of Three Dimensional Images, 3D-Forum Society 13, 3 (Sept.), 66–70.]]Google Scholar
24. HOCHBAUM, D. S., AND SCHMOYS, D. B. 1985. A best possible heuristic for the k-center problem. Mathematics of Operations Research 10, 2 (May), 180–184.]]Google ScholarDigital Library
25. ITU-R. 1994. Methods for subjective assessment of small impairments in audio systems including multichannel sound systems, ITU-R BS 1116.]]Google Scholar
26. LAGRANGE, M., AND MARCHAND, S. 2001. Real-time additive synthesis of sound by taking advantage of psychoacoustics. In Proceedings of the COST G-6 Conference on Digital Audio Effects (DAFX-01), Limerick, Ireland, December 6–8.]]Google Scholar
27. LARSSON, P., VÄSTFJÄLL, D., AND KLEINER, M. 2002. Better presence and performance in virtual environments by improved binaural sound rendering. proceedings of the AES 22nd Intl. Conf. on virtual, synthetic and entertainment audio, Espoo, Finland (June), 31–38.]]Google Scholar
28. LIKAS, A., VLASSIS, N., AND VERBEEK, J. 2003. The global k-means clustering algorithm. Pattern Recognition 36, 2, 451–461.]]Google ScholarCross Ref
29. LOKKI, T., GRÖHN, M., SAVIOJA, L., AND TAKALA, T. 2000. A case study of auditory navigation in virtual acoustic environments. Proceedings of Intl. Conf. on Auditory Display (ICAD2000).]]Google Scholar
30. MARTENS, W. 1987. Principal components analysis and resynthesis of spectral cues to perceived direction. In Proc. Int. Computer Music Conf. (ICMC’87), 274–281.]]Google Scholar
31. MOORE, B. C. J., GLASBERG, B., AND BAER, T. 1997. A model for the prediction of thresholds, loudness and partial loudness. J. of the Audio Engineering Society 45, 4, 224–240. Software available at http://hearing.psychol.cam.ac.uk/Demos/demos.html.]]Google Scholar
32. MOORE, B. C. 1997. An introduction to the psychology of hearing. Academic Press, 4th edition.]]Google Scholar
33. PAINTER, E. M., AND SPANIAS, A. S. 1997. A review of algorithms for perceptual coding of digital audio signals. DSP-97.]]Google Scholar
34. PAQUETTE, E., POULIN, P., AND DRETTAKIS, G. 1998. A light hierarchy for fast rendering of scenes with many lights. Proceedings of EUROGRAPHICS ’98.]]Google ScholarCross Ref
35. PIERCE, A. 1984. Acoustics. An introduction to its physical principles and applications. 3rd edition, American Institute of Physics.]]Google Scholar
36. SAVIOJA, L., HUOPANIEMI, J., LOKKI, T., AND VÄÄNÄNEN, R. 1999. Creating interactive virtual acoustic environments. J. of the Audio Engineering Society 47, 9 (Sept.), 675–705.]]Google Scholar
37. SENSAURA, 2001. ZoomFX, MacroFX, Sensaura©. http://www.sensaura.co.uk.]]Google Scholar
38. SOUNDBLASTER, 2004. Creative Labs Soundblaster©. http://www.soundblaster.com.]]Google Scholar
39. STEIGLITZ, K. 1996. A DSP Primer with applications to digital audio and computer music. Addison Wesley.]] Google ScholarDigital Library
40. TSINGOS, N., AND GASCUEL, J.-D. 1997. Soundtracks for computer animation: sound rendering in dynamic environments with occlusions. Proceedings of Graphics Interface ’97 (May), 9–16.]] Google ScholarDigital Library
41. TSINGOS, N., FUNKHOUSER, T., NGAN, A., AND CARLBOM, I. 2001. Modeling acoustics in virtual environments using the uniform theory of diffraction. ACM Computer Graphics, SIGGRAPH’01 Proceedings (Aug.), 545–552.]] Google ScholarDigital Library
42. VAN DEN DOEL, K., PAI, D. K., ADAM, T., KORTCHMAR, L., AND PICHORA-FULLER, K. 2002. Measurements of perceptual quality of contact sound models. In Proceedings of the International Conference on Auditory Display (ICAD 2002), Kyoto, Japan, 345–349.]]Google Scholar
43. VAN DEN DOEL, K., KNOTT, D., AND PAI, D. K. 2004. Interactive simulation of complex audio-visual scenes. Presence: Teleoperators and Virtual Environments 13, 1.]] Google ScholarDigital Library
44. VROOMEN, J., ANDDE GELDER, B. 2004. Perceptual effects of cross-modal stimulation: Ventriloquism and the freezing phenomenon. In Handbook of multisensory processes, G. Calvert, C. Spence, and B. E. Stein, Eds. M.I.T. Press.]]Google Scholar
45. WENZEL, E., MILLER, J., AND ABEL, J. 2000. A software-based system for interactive spatial sound synthesis. Proceeding of ICAD 2000, Atlanta, USA (April).]]Google Scholar
46. ZWICKER, E., AND FASTL, H. 1999. Psychoacoustics: Facts and Models. Springer. Second Upadated Edition.]] Google ScholarDigital Library