“Technologies for augmented reality systems: realizing ultrasound-guided needle biopsies” by State, Livingston, Garrett, Hirota, Whitton, et al. …

  • ©Andrei State, Mark A. Livingston, William F. Garrett, Gentaro Hirota, Mary C. Whitton, Etta D. Pisano, and Henry Fuchs

Conference:


Type:


Title:

    Technologies for augmented reality systems: realizing ultrasound-guided needle biopsies

Presenter(s)/Author(s):


Abstract:


    We present a real-time stereoscopic video-see-through augmented reality (AR) system applied to the medical procedure known as ultrasound-guided needle biopsy of the breast. The AR system was used by a physician during procedures on breast models and during non-invasive examinations of human subjects. The system merges rendered live ultrasound data and geometric elements with stereo images of the patient acquired through head-mounted video cameras and presents these merged images to the physician in a head-mounted display. The physician sees a volume visualization of the ultrasound data directly under the ultrasound probe, properly registered within the patient and with the biopsy needle. Using this system, a physician successfully guided a needle into an artificial tumor within a training phantom of a human breast. We discuss the construction of the AR system and the issues and decisions which led to the system architecture and the design of the video see-through head-mounted display. We designed methods to properly resolve occlusion of the real and synthetic image elements. We developed techniques for real-time volume visualization of time- and position-varying ultrasound data. We devised a hybrid tracking system which achieves improved registration of synthetic and real imagery and we improved on previous techniques for calibration of a magnetic tracker.

References:


    1. AZUMA, R. A Survey of Augmented Reality. SIGGRAPH 1995 Course Notes #9 (Developing Advanced Virtual Reality Applications), pp. 20-1 through 20-38.
    2. AZUMA, R., BISHOP, G. Improving Static and Dynamic Registration in an Optical See-through HMD. Proceedings of SIGGRAPH 94 (Orlando, Florida, July 24-29, 1994). In Computer Graphics Proceedings, Annual Conference Series, 1994, ACM SIGGRAPH, pp. 197-204. 
    3. BAJURA, M., FUCHS, H., OHBUCHI, R. Merging Virtual Objects with the Real World. Proceedings of SIGGRAPH ’92 (Chicago, Illinois, July 26- 31, 1992). In Computer Graphics 26, 2 (July 1992), ACM SIGGRAPH, New York, 1992, pp. 203-210. 
    4. BAJURA, M., NEUMANN, U. Dynamic Registration Correction in Video- Based Augmented Reality Systems. IEEE Computer Graphics and Applications (September, 1995), pp. 52-60. 
    5. BARBER, C.B., DOBKIN, D.P., HUHDANPAA, H. The Quickhull Algorithm for Convex Hull. Geometry Center Technical Report GCG53, 1993.
    6. BRYSON, S. Measurement and Calibration of Static Distortion of Position Data from 3D Trackers. SPIE Stereoscopic Displays and Applications III, 1992, pp. 244-255.
    7. CABRAL, B., CAM, N., FORAN, J. Accelerated Volume Rendering and Tomographic Reconstruction Using Texture Mapping Hardware. Proceedings of the 1994 Symposium on Volume Visualization (Washington, D. C., October 1994), pp. 91-97, 131, back cover. 
    8. COLUCCI, D., CHI, V. Computer Glasses: A Compact, Lightweight, and Cost-effective Display for Monocular and Tiled Wide Field-of-View Systems. Proceedings ofSPIE, Vol. 2537, pp. 61-70 (1995).
    9. CULLIP, T.J., NEUMANN, U. Accelerating Volume Reconstruction With 3D Texture Hardware. University of North Carolina Department of Computer Science Technical Report TR93-027. 
    10. FUCHS, H., ABRAM, G., GRANT, E. Near Real-time Shaded Display of Rigid Objects. Proceedings of SIGGRAPH ’83 (July 1983). In Computer Graphics; 17, 3 (July 1983), ACM SIGGRAPH, New York, 1983, pp. 65- 72. 
    11. FUCHS, H., KEDEM, Z., NAYLOR, B. On Visible Surface Generation by a Priori Tree Structures. Proceedings of SIGGRAPH ’80 (July 1980). In Computer Graphics; 14, 3 (July 1980), ACM SIGGRAPH, New York, 1980, pp. 124-133. 
    12. FURNESS, T. The super cockpit and its human factors challenges. Proceedings of the Human Factors Society, 30, 1968, pp. 48-52.
    13. GARRETT, W.F., FUCHS, H., STATE, A., WHITTON, M.C. Real-Time Incremental Visualization of Dynamic Ultrasound Volumes Using Parallel BSP Trees. University of North Carolina Department of Computer Science Technical Report TR96-018. 
    14. GHAZlSAEDY, M., ADAMCZYK, D., SANDIN, D., KENYON, R., DEFANTI, T. Ultrasonic Calibration of a Magnetic Tracker in a Virtual Reality Space. Proceedings of the Virtual Reality Annual International Symposium ’95, IEEE Computer Society, Los Alamitos, CA, 1995, pp. 179-188. 
    15. LALOUCHE, R.C., BICKMORE, D., TESSLER, F., MANKOVICH, H.K., KANGARALOO, H. Three-dimensional reconstruction of ultrasound images. SPIE’89, Medical Imaging, SPIE, 1989, pp. 59-66.
    16. LIVINGSTON, M.A., STATE, A. Magnetic Tracker Calibration for Improved Registration in Augmented Reality Systems. University of North Carolina Department of Computer Science Technical Report TR95- 037.
    17. LORENSEN, W., CLINE, H., NAFIS, C., KIKINIS, R., ALTOBELLI, D., GLEASON, L. Enhancing reality in the operating room. Proceedings of Visualization 1993 (Los Alamitos, CA, October 1993), pp. 410-415.
    18. MELLOR, J.P. Enhanced Reality Visualization in a Surgical Environment. MS Thesis, MIT Department of Electrical Engineering, 1995.
    19. NAYLOR, B., AMANATIDES, J., THIBAULT, W. Merging BSP Tree Yields Polyhedral Set Operations. Proceedings of SIGGRAPH ’90 (Dallas, Texas, August 1990). In Computer Graphics; 24, 4 (August 1990), pp. 115-124. 
    20. NELSON, T., ELVINS, T. Visualization of 3D Ultrasound Data. IEEE Computer Graphics and Applications, (November, 1993), pp. 50-57. 
    21. OHBUCHI, R., CHEN, D.T., FUCHS, H.Incremental Volume Reconstruction and Rendering for 3D Ultrasound Imaging. University of North Carolina Department of Computer Science Technical Report TR92- 037. 
    22. ROBINETT, W., ROLLAND, J.P. A Computational Model for the Stereoscopic Optics of a Head-Mounted Display. Presence, Vol. 1, No. 1 (Winter 1992), pp. 45-62. 
    23. ROLLAND, J.P., BIOCCA, F., BARLOW, T., KANCHERLA, A.R. Quantification of Adaptation to Virtual-Eye Location in See-Thru Head- Mounted Displays. Proceedings of the Virtual Reality: Annual International Symposium ’95, IEEE Computer Society, Los Alamitos, CA, 1995, pp. 56-66. 
    24. ROSE, E., BREEN, D., AHLERS, K., CRAMPTON, C., TUCERYAN, M., WHITAKER, R., GREER, D. Annotating Real-World Objects Using Augmented Reality. Proceedings of Computer Graphics International 1995 (Leeds, UK, June, 25-30, 1995), pp. 357-370. 
    25. SIMS, D. New Realities in Aircraft Design and Manufacture. IEEE Computer Graphics and Applications, (March, 1994), p. 91. 
    26. STATE, A., CHEN, D.T., TECTOR, C., BRANDT, A., CHEN, H., OHBUCHI, R., BAJURA, M., FUCHS, H. Case Study: Observing a Volume-Rendered Fetus within a Pregnant Patient. Proceedings of IEEE Visualization ’94, October 1994, pp. 364-368, CP-41. 
    27. STATE, A., HIROTA, G., CHEN, D.T., GARRETT, W.F., LIVINGSTON, M.A. Superior Augmented Reality Registration by Integrating Landmark Tracking and Magnetic Tracking. Proceedings of SIGGRAPH ’96 (New Orleans, LA, August 4-9, 1996). In Computer Graphics Proceedings, Annual Conference Series, 1996, ACM SIGGRAPH. 
    28. STATE, A., MCALLISTER, J., NEUMANN, U., CHEN, H., CULLIP, T.J., CHEN, D.T., FUCHS, H. Interactive Volume Visualization on a Heterogeneous Message-Passing Multicomputer. Proceedings of the 1995 Symposium on Interactive 3D Graphics (Monterey, CA, April 1995), pp. 69-74, 208, front cover. 
    29. SUTHERLAND, I. A head-mounted three dimensional display. Proceedings of the Fall Joint Computer Conference, 1968.
    30. WLOKA, M.M., ANDERSON, B.G. Resolving Occlusion in Augmented Reality. Proceedings of the 1995 Symposium on Interactive 3D Graphics (Monterey, CA, April 1995), pp. 5-12. 

ACM Digital Library Publication:


Overview Page: