“Perceptual audio rendering of complex virtual environments” by Tsingos, Gallo and Drettakis

  • ©Nicolas Tsingos, Emmanuel Gallo, and George Drettakis




    Perceptual audio rendering of complex virtual environments



    We propose a real-time 3D audio rendering pipeline for complex virtual scenes containing hundreds of moving sound sources. The approach, based on auditory culling and spatial level-of-detail, can handle more than ten times the number of sources commonly available on consumer 3D audio hardware, with minimal decrease in audio quality. The method performs well for both indoor and outdoor environments. It leverages the limited capabilities of audio hardware for many applications, including interactive architectural acoustics simulations and automatic 3D voice management for video games.Our approach dynamically eliminates inaudible sources and groups the remaining audible sources into a budget number of clusters. Each cluster is represented by one impostor sound source, positioned using perceptual criteria. Spatial audio processing is then performed only on the impostor sound sources rather than on every original source thus greatly reducing the computational cost.A pilot validation study shows that degradation in audio quality, as well as localization impairment, are limited and do not seem to vary significantly with the cluster budget. We conclude that our real-time perceptual audio rendering pipeline can generate spatialized audio for complex auditory environments without introducing disturbing changes in the resulting perceived soundfield.


    1. ANSI. 1978. American national standard method for the calculation of the absorption of sound by the atmosphere. ANSI S1.26-1978, American Institute of Physics (for Acoustical Society of America), New York.]]Google Scholar
    2. BEGAULT, D. R., MCCLAIN, B. U., AND ANDERSON, M. R. 2001. Early reflection thresholds for virtual sound sources. In Proc. 2001 Int. Workshop on Spatial Media.]]Google Scholar
    3. BEGAULT, D. R. 1994. 3D Sound for Virtual Reality and Multimedia. Academic Press Professional.]] Google ScholarDigital Library
    4. BLAUERT, J. 1983. Spatial Hearing: The Psychophysics of Human Sound Localization. M.I.T. Press, Cambridge, MA.]]Google Scholar
    5. BORISH, J. 1984. Extension of the image model to arbitrary polyhedra. J. of the Acoustical Society of America 75, 6.]]Google ScholarCross Ref
    6. BRANDENBURG, K. 1999. mp3 and AAC explained. AES 17th International Conference on High-Quality Audio Coding (Sept.).]]Google Scholar
    7. BREGMAN, A. 1990. Auditory Scene Analysis, The perceptual school of sound. The MIT Press.]]Google Scholar
    8. CHEN, J., VEEN, B. V., AND HECOX, K. 1995. A spatial feature extraction and regularization model for the head-related transfer function. J. of the Acoustical Society of America 97 (Jan.), 439–452.]]Google ScholarCross Ref
    9. CHEN, H., WALLACE, G., GUPTA, A., LI, K., FUNKHOUSER, T., AND COOK, P. 2002. Experiences with scalability of display walls. Proceedings of the Immersive Projection Technology (IPT) Workshop (Mar.).]]Google Scholar
    10. DIRECT SOUND 3D, 2004. Direct X homepage, Microsoft©. http://www.microsoft.com/windows/directx/default.asp.]]Google Scholar
    11. DOBASHI, Y., YAMAMOTO, T., AND NISHITA, T. 2003. Real-time rendering of aerodynamic sound using sound textures based on computational fluid dynamics. ACM Transactions on Graphics 22, 3 (Aug.), 732–740. (Proceedings of ACM SIGGRAPH 2003).]] Google ScholarDigital Library
    12. EAX, 2004. Environmental audio extensions 4.0, Creative©. http://www.soundblaster.com/eaudio.]]Google Scholar
    13. ELLIS, D. 1992. A perceptual representation of audio. Master’s thesis, Massachusets Institute of Technology.]]Google Scholar
    14. FALLER, C., AND BAUMGARTE, F. 2002. Binaural cue coding applied to audio compression with flexible rendering. In Proc. 113th AES Convention.]]Google Scholar
    15. FILIPANITS, F. 1994. Design and implementation of an auralization system with a spectrum-based temporal processing optimization. Master thesis, Univ. of Miami.]]Google Scholar
    16. FOUAD, H., HAHN, J., AND BALLAS, J. 1997. Perceptually based scheduling algorithms for real-time synthesis of complex sonic environments. proceedings of the 1997 International Conference on Auditory Display (ICAD’97), Xerox Palo Alto Research Center, Palo Alto, USA.]]Google Scholar
    17. FOUAD, H., BALLAS, J., AND BROCK, D. 2000. An extensible toolkit for creating virtual sonic environments. Proceedings of Intl. Conf. on Auditory Display (Atlanta, USA, May 2000).]]Google Scholar
    18. FUNKHOUSER, T., AND SEQUIN, C. 1993. Adaptive display algorithms for interactive frame rates during visualization of complex virtual environments. Computer Graphics (SIGGRAPH ’93 proceedings), Los Angeles, CA (August), 247–254.]] Google ScholarDigital Library
    19. FUNKHOUSER, T., MIN, P., AND CARLBOM, I. 1999. Real-time acoustic modeling for distributed virtual environments. ACM Computer Graphics, SIGGRAPH ’99 Proceedings (Aug.), 365–374.]] Google ScholarDigital Library
    20. GARDNER, W. 1997. Reverberation algorithms. In Applications of Digital Signal Processing to Audio and Acoustics, M. Kahrs and K. Brandenburg, Eds. Kluwer Academic Publishers, 85–131.]]Google Scholar
    21. GREWIN, C. 1993. Methods for quality assessment of low bit-rate audio codecs. proceedings of the 12th AES conference, 97–107.]]Google Scholar
    22. HERDER, J. 1999. Optimization of sound spatialization resource management through clustering. The Journal of Three Dimensional Images, 3D-Forum Society 13, 3 (Sept.), 59–65.]]Google Scholar
    23. HERDER, J. 1999. Visualization of a clustering algorithm of sound sources based on localization errors. The Journal of Three Dimensional Images, 3D-Forum Society 13, 3 (Sept.), 66–70.]]Google Scholar
    24. HOCHBAUM, D. S., AND SCHMOYS, D. B. 1985. A best possible heuristic for the k-center problem. Mathematics of Operations Research 10, 2 (May), 180–184.]]Google ScholarDigital Library
    25. ITU-R. 1994. Methods for subjective assessment of small impairments in audio systems including multichannel sound systems, ITU-R BS 1116.]]Google Scholar
    26. LAGRANGE, M., AND MARCHAND, S. 2001. Real-time additive synthesis of sound by taking advantage of psychoacoustics. In Proceedings of the COST G-6 Conference on Digital Audio Effects (DAFX-01), Limerick, Ireland, December 6–8.]]Google Scholar
    27. LARSSON, P., VÄSTFJÄLL, D., AND KLEINER, M. 2002. Better presence and performance in virtual environments by improved binaural sound rendering. proceedings of the AES 22nd Intl. Conf. on virtual, synthetic and entertainment audio, Espoo, Finland (June), 31–38.]]Google Scholar
    28. LIKAS, A., VLASSIS, N., AND VERBEEK, J. 2003. The global k-means clustering algorithm. Pattern Recognition 36, 2, 451–461.]]Google ScholarCross Ref
    29. LOKKI, T., GRÖHN, M., SAVIOJA, L., AND TAKALA, T. 2000. A case study of auditory navigation in virtual acoustic environments. Proceedings of Intl. Conf. on Auditory Display (ICAD2000).]]Google Scholar
    30. MARTENS, W. 1987. Principal components analysis and resynthesis of spectral cues to perceived direction. In Proc. Int. Computer Music Conf. (ICMC’87), 274–281.]]Google Scholar
    31. MOORE, B. C. J., GLASBERG, B., AND BAER, T. 1997. A model for the prediction of thresholds, loudness and partial loudness. J. of the Audio Engineering Society 45, 4, 224–240. Software available at http://hearing.psychol.cam.ac.uk/Demos/demos.html.]]Google Scholar
    32. MOORE, B. C. 1997. An introduction to the psychology of hearing. Academic Press, 4th edition.]]Google Scholar
    33. PAINTER, E. M., AND SPANIAS, A. S. 1997. A review of algorithms for perceptual coding of digital audio signals. DSP-97.]]Google Scholar
    34. PAQUETTE, E., POULIN, P., AND DRETTAKIS, G. 1998. A light hierarchy for fast rendering of scenes with many lights. Proceedings of EUROGRAPHICS ’98.]]Google ScholarCross Ref
    35. PIERCE, A. 1984. Acoustics. An introduction to its physical principles and applications. 3rd edition, American Institute of Physics.]]Google Scholar
    36. SAVIOJA, L., HUOPANIEMI, J., LOKKI, T., AND VÄÄNÄNEN, R. 1999. Creating interactive virtual acoustic environments. J. of the Audio Engineering Society 47, 9 (Sept.), 675–705.]]Google Scholar
    37. SENSAURA, 2001. ZoomFX, MacroFX, Sensaura©. http://www.sensaura.co.uk.]]Google Scholar
    38. SOUNDBLASTER, 2004. Creative Labs Soundblaster©. http://www.soundblaster.com.]]Google Scholar
    39. STEIGLITZ, K. 1996. A DSP Primer with applications to digital audio and computer music. Addison Wesley.]] Google ScholarDigital Library
    40. TSINGOS, N., AND GASCUEL, J.-D. 1997. Soundtracks for computer animation: sound rendering in dynamic environments with occlusions. Proceedings of Graphics Interface ’97 (May), 9–16.]] Google ScholarDigital Library
    41. TSINGOS, N., FUNKHOUSER, T., NGAN, A., AND CARLBOM, I. 2001. Modeling acoustics in virtual environments using the uniform theory of diffraction. ACM Computer Graphics, SIGGRAPH’01 Proceedings (Aug.), 545–552.]] Google ScholarDigital Library
    42. VAN DEN DOEL, K., PAI, D. K., ADAM, T., KORTCHMAR, L., AND PICHORA-FULLER, K. 2002. Measurements of perceptual quality of contact sound models. In Proceedings of the International Conference on Auditory Display (ICAD 2002), Kyoto, Japan, 345–349.]]Google Scholar
    43. VAN DEN DOEL, K., KNOTT, D., AND PAI, D. K. 2004. Interactive simulation of complex audio-visual scenes. Presence: Teleoperators and Virtual Environments 13, 1.]] Google ScholarDigital Library
    44. VROOMEN, J., ANDDE GELDER, B. 2004. Perceptual effects of cross-modal stimulation: Ventriloquism and the freezing phenomenon. In Handbook of multisensory processes, G. Calvert, C. Spence, and B. E. Stein, Eds. M.I.T. Press.]]Google Scholar
    45. WENZEL, E., MILLER, J., AND ABEL, J. 2000. A software-based system for interactive spatial sound synthesis. Proceeding of ICAD 2000, Atlanta, USA (April).]]Google Scholar
    46. ZWICKER, E., AND FASTL, H. 1999. Psychoacoustics: Facts and Models. Springer. Second Upadated Edition.]] Google ScholarDigital Library

ACM Digital Library Publication: