“Practical aspects of spectral data in digital content production” Chaired by – ACM SIGGRAPH HISTORY ARCHIVES

“Practical aspects of spectral data in digital content production” Chaired by

  • ©

Conference:


Type(s):


Entry Number: 11

Title:

    Practical aspects of spectral data in digital content production

Presenter(s)/Author(s):


Abstract:


    Compared to path tracing, spectral rendering is still often considered to be a niche application used mainly to produce optical wave effects like dispersion or diffraction. And while over the last years more and more people started exploring the potential of spectral image synthesis, it is still widely assumed to be only of importance in high-quality offline applications associated with long render times and high visual fidelity.

    While it is certainly true that describing light interactions in a spectral way is a necessity for predictive rendering, its true potential goes far beyond that. Used correctly, not only will it guarantee colour fidelity, but it will also simplify workflows for all sorts of applications.

    Wētā Digital’s renderer Manuka showed that there is a place for a spectral renderer in a production environment and how workflows can be simplified if the whole pipeline adapts. Picking up from the course last year, we want to continue the discussion we started as we firmly believe that spectral data is the future in content production. The authors feel enthusiastic about more people being aware of the advantages that spectral rendering and spectral workflows bring and share the knowledge we gained over many years. The novel workflows emerged during the adaptation of spectral techniques at a number of large companies are introduced to a wide audience including technical directors, artists and researchers. However, while last year’s course concentrated primarily on the algorithmic sides of spectral image synthesis, this year we want to focus on the practical aspects.

    We will draw examples from virtual production, digital humans over spectral noise reduction to image grading, therefore showing the usage of spectral data enhancing each and every single part of the image pipeline.

References:


    1. Catherine Cooksey, David Allen, and Benjamin Tsai. 2017. Reference Data Set of Human Skin Reflectance.?
    2. Weta Digital. 2022.?Physlight.?Physlight
    3. R Donaldson. 1954. Spectrophotometry of fluorescent pigments.?British Journal of Applied Physics?5, 6 (jun 1954), 210–214.?
    4. Luca Fascione, Johannes Hanika, Mark Leone, Marc Droske, Jorge Schwarzhaupt, Tomas Davidovic, Andrea Weidlich, and Johannes Meng. 2018. Manuka: A Batch-Shading Architecture for Spectral Path Tracing in Movie Production.?ACM Transactions on Graphics?37 (08 2018), 1–18.?
    5. Alban Fichet, Romain Pacanowski, and Alexander Wilkie. 2021. An OpenEXR Layout for Spectral images.?Journal of Computer Graphics Techniques (JCGT)?10, 3 (29 September 2021), 1–18. http://jcgt.org/published/0010/03/01/
    6. Qingqin Hua, Alban Fichet, and Alexander Wilkie. 2021. A Compact Representation for Fluorescent Spectral Data. In?Eurographics Symposium on Rendering – DL-only Track, Adrien Bousseau and Morgan McGuire (Eds.). The Eurographics Association.?
    7. Longqian Huang, Ruichen Luo, Xu Liu, and Xiang Hao. 2022. Spectral imaging with deep learning.?Light: Science & Applications?11 (03 2022), 61.?
    8. Wenzel Jakob and Johannes Hanika. 2019. A Low-Dimensional Function Space for Efficient Spectral Upsampling.?Computer Graphics Forum (Proceedings of Eurographics)?38, 2 (March 2019).
    9. Alisa Jung, A. Wilkie, J. Hanika, W. Jakob, and C. Dachsbacher. 2019. Wide Gamut Spectral Upsampling with Fluorescence.?Computer Graphics Forum?38 (07 2019), 87–96.?
    10. Lars K?nig, Alisa Jung, and Carsten Dachsbacher. 2020. Improving Spectral Upsampling with Fluorescence. In?Workshop on Material Appearance Modeling, Reinhard Klein and Holly Rushmeier (Eds.). The Eurographics Association.?
    11. Tom Lister, Philip A. Wright, and Paul H. Chappell. 2012. Optical properties of human skin.?Journal of Biomedical Optics?17, 9 (2012), 1 — 15.?
    12. Ian Mallett and Cem Yuksel. 2019. Spectral Primary Decomposition for Rendering with sRGB Reflectance. In?Eurographics Symposium on Rendering – DL-only and Industry Track, Tamy Boubekeur and Pradeep Sen (Eds.). The Eurographics Association.?
    13. Johannes Meng, Florian Simon, Johannes Hanika, and Carsten Dachsbacher. 2015. Physically meaningful rendering using tristimulus colours. In?Computer Graphics Forum, Vol. 34. Wiley Online Library, 31–40.
    14. Arash Mirhashemi. 2018. Introducing spectral moment features in analyzing the SpecTex hyperspectral texture database.?Machine Vision and Applications?29 (04 2018).?
    15. Hisanari Otsu, Masafumi Yamamoto, and Toshiya Hachisuka. 2018. Reproducing spectral reflectances from tristimulus colours. In?Computer Graphics Forum, Vol. 37. Wiley Online Library, 370–381.
    16. Mark S. Peercy. 1993. Linear Color Representations for Full Speed Spectral Rendering. In?Proceedings of the 20th Annual Conference on Computer Graphics and Interactive Techniques?(Anaheim, CA)?(SIGGRAPH ’93).?Association for Computing Machinery, New York, NY, USA, 191–198.?
    17. Scyllarus. 2016.?HSZ Data Format.?https://scyllarus.data61.csiro.au/data/hsz-data-format/
    18. Brian Smits. 1999. An RGB-to-spectrum conversion for reflectances.?Journal of Graphics Tools?4, 4 (1999), 11–22.
    19. Yinlong Sun, M.S. Drew, and F.D. Fracchia. 1999. Representing spectral functions by a composite model of smooth and spiky components for efficient full-spectrum photorealism. In?Proceedings Workshop on Photometric Modeling for Computer Vision and Graphics (Cat. No.PR00271).?4–11.?
    20. Lucia T?dov?, Alexander Wilkie, and Luca Fascione. 2021. Moment-based Constrained Spectral Uplifting. In?Eurographics Symposium on Rendering – DL-only Track, Adrien Bousseau and Morgan McGuire (Eds.). The Eurographics Association.?
    21. Mark van de Ruit and Elmar Eisemann. 2021. A multi-pass method for accelerated spectral sampling.?Computer Graphics Forum?40 (2021).
    22. Andrea Weidlich, Alex Forsythe, Scott Dyer, Thomas Mansencal, Johannes Hanika, Alexander Wilkie, Luke Emrose, and Anders Langlands. 2021. Spectral Imaging in Production: Course Notes Siggraph 2021. In?ACM SIGGRAPH 2021 Courses?(Virtual Event, USA)?(SIGGRAPH ’21).?Association for Computing Machinery, New York, NY, USA, Article 14, 90 pages.?
    23. Rex West, Iliyan Georgiev, Adrien Gruson, and Toshiya Hachisuka. 2020. Continuous Multiple Importance Sampling.?ACM Trans. Graph.?39, 4, Article 136 (jul 2020), 12 pages.?
    24. A. Wilkie, S. Nawaz, M. Droske, A. Weidlich, and J. Hanika. 2014. Hero Wavelength Spectral Sampling. In?Proceedings of the 25th Eurographics Symposium on Rendering?(Lyon, France)?(EGSR ’14).?Eurographics Association, Goslar, DEU, 123–131.
    25. Alexander Wilkie, Robert Tobler, and Werner Purgathofer. 2000. Raytracing of Dispersion Effects in Transparent Materials.
    26. Chris Wyman, Peter-Pike Sloan, and Peter Shirley. 2013. Simple Analytic Approximations to the CIE XYZ Color Matching Functions.?Journal of Computer Graphics Techniques (JCGT)?2, 2 (12 July 2013), 1–11. http://jcgt.org/published/0002/02/01/
    27. X-Rite. 2022.?AxF Data Format.?https://www.xrite.com/appearance-exchange-format-axf
    28. [n.d.]. In-camera VFX Camera Color Calibration. https://docs.unrealengine.com/4.27/en-US/WorkingWithMedia/IntegratingMedia/InCameraVFX/InCameraVFXCameraCalibration/
    29. Carlos F Borges. 1991. Trichromatic approximation for computer graphics illumination models. In?Proceedings of the 18th annual conference on Computer graphics and interactive techniques (SIGGRAPH ’91).?101–104.
    30. Paul Debevec. 1998. Rendering Synthetic Objects into Real Scenes: Bridging Traditional and Image-Based Graphics with Global Illumination and High Dynamic Range Photography. In?Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’98).?Association for Computing Machinery, New York, NY, USA, 189–198.?
    31. Paul Debevec, Andreas Wenger, Chris Tchou, Andrew Gardner, Jamie Waese, and Tim Hawkins. 2002. A Lighting Reproduction Approach to Live-Action Compositing.?ACM Trans. Graph.?21, 3 (Jul 2002), 547–556.?
    32. Jonathan Erland. 2011. Chromatic Chaos: Implications of Newly Introduced Forms of Stage-light. National Association of Broadcasters (NAB) Conference.
    33. Nahum Gat and CA Torrance. 1998. Real-time multi-and hyper-spectral imaging for remote sensing and machine vision: an overview. In?Proc. 1998 ASAE Annual International Mtg.?Citeseer.
    34. Pierre-Lo?c Hamon, James Harmer, Stuart Penn, and Nicolas Scapel. 2014. Gravity: Motion control and face integration. In?ACM SIGGRAPH 2014 Talks.?1–1.
    35. Jay Holben. 2020. The Mandalorian: This Is the Way. https://ascmag.com/articles/the-mandalorian
    36. Jack Holm, Tom Maier, Paul Debevec, Chloe LeGendre, Joshua Pines, Jonathan Erland, George Joblove, Scott Dyer, Blake Sloan, Joe di Gennaro, et al. 2016. A cinematographic spectral similarity index. In?SMPTE 2016 Annual Technical Conference and Exhibition.?SMPTE, 1–36.
    37. Jun Jiang, Dengyu Liu, Jinwei Gu, and Sabine S?sstrunk. 2013. What is the space of spectral sensitivity functions for digital color cameras?. In?2013 IEEE Workshop on Applications of Computer Vision (WACV).?IEEE, 168–179.
    38. Noah Kadner. 2021a. 1899 Wraps Innovative Virtual Production. https://ascmag.com/articles/1899-wraps-virtual-production
    39. Noah Kadner. 2021b. Color Fidelity in LED Volumes. https://ascmag.com/articles/color-fidelity-in-led-volumes
    40. Noah Kadner. 2021c. On The Walls: Virtual Production for Series Shooting. https://ascmag.com/articles/on-the-walls
    41. Chloe LeGendre, Lukas Lepicovsky, and Paul Debevec. 2022. Jointly Optimizing Color Rendition and In-Camera Backgrounds in an RGB Virtual Production Stage.?(arXiv.org)?(2022).
    42. Chloe LeGendre, Xueming Yu, and Paul Debevec. 2017. Optimal LED selection for multi-spectral lighting reproduction.?Electronic Imaging?2017, 8 (2017), 25–32.
    43. Chloe LeGendre, Xueming Yu, Dai Liu, Jay Busch, Andrew Jones, Sumanta Pattanaik, and Paul Debevec. 2016. Practical multispectral lighting reproduction.?ACM Transactions on Graphics (TOG)?35, 4 (2016), 1–11.
    44. Calvin S McCamy, Harold Marcus, James G Davidson, et al. 1976. A color-rendition chart.?J. App. Photog. Eng?2, 3 (1976), 95–99.
    45. Muhammad Uzair, Arif Mahmood, Faisal Shafait, Christian Nansen, and Ajmal Mian. 2015. Is spectral reflectance of the face a reliable biometric??Optics express?23, 12 (2015), 15160–15173.
    46. Andrea Weidlich, Alex Forsythe, Scott Dyer, Thomas Mansencal, Johannes Hanika, Alexander Wilkie, Luke Emrose, and Anders Langlands. 2021. Spectral imaging in production: course notes Siggraph 2021. In?ACM SIGGRAPH 2021 Courses.?1–90.
    47. Andreas Wenger, Tim Hawkins, and Paul Debevec. 2003. Optimizing color matching in a lighting reproduction system for complex subject and illuminant spectra. In?Rendering Techniques.?Citeseer, 249–259.
    48. Carlos Aliaga, Christophe Hery, and Mengqi Xia. 2022. Estimation of Spectral Biophysical Skin Properties from Captured RGB Albedo.
    49. Sarah Alotaibi and William AP Smith. 2017. A biophysical 3D morphable model of face appearance. In?Proceedings of the IEEE International Conference on Computer Vision Workshops.?824–832.
    50. Gladimir VG Baranoski and Aravind Krishnaswamy. 2010.?Light and skin interactions: simulations for computer graphics applications.?Morgan Kaufmann.
    51. Alexey N Bashkatov, Elina A Genina, and Valery V Tuchin. 2011. Optical properties of skin, subcutaneous, and muscle tissues: a review.?Journal of Innovative Optical Health Sciences?4, 01 (2011), 9–38.
    52. Benedikt Bitterli, Srinath Ravichandran, Thomas M?ller, Magnus Wrenninge, Jan Nov?k, Steve Marschner, and Wojciech Jarosz. 2018. A Radiative Transfer Framework for Non-Exponential Media.?ACM Trans. Graph.?37, 6, Article 225 (Dec. 2018), 17 pages.
    53. Tenn F Chen, Gladimir VG Baranoski, Bradley W Kimmel, and Erik Miranda. 2015. Hyper-spectral modeling of skin appearance.?ACM Transactions on Graphics (TOG)?34, 3 (2015), 1–14.
    54. Blender Online Community. 2020. Blender: a 3D modelling and rendering package. (2020). http://www.blender.org
    55. Carolin Czekalla, Karl Heinz Sch?nborn, J?rgen Lademann, and Martina C Meinke. 2019. Noninvasive determination of epidermal and stratum corneum thickness in vivo using two-photon microscopy and optical coherence tomography: Impact of body area, age, and gender.?Skin pharmacology and physiology?32, 3 (2019), 142–150.
    56. Philip E Dennison, Kerry Q Halligan, and Dar A Roberts. 2004. A comparison of error metrics and constraints for multiple endmember spectral mixture analysis and spectral angle mapper.?Remote Sensing of Environment?93, 3 (2004), 359–367.
    57. Craig Donner and Henrik Wann Jensen. 2006. A Spectral BSSRDF for Shading Human Skin.?Rendering techniques?2006 (2006), 409–418.
    58. Eugene d’ Eon. 2019. A Reciprocal Formulation of Nonexponential Radiative Transfer. 2: Monte Carlo Estimation and Diffusion Approximation.?Journal of Computational and Theoretical Transport?48, 6 (Sep 2019), 201–262.
    59. Eugene d’Eon. 2021. A Hitchhiker’s Guide to Multiple Scattering. (2021).
    60. Thomas B Fitzpatrick. 1988. The validity and practicality of sun-reactive skin types I through VI.?Archives of dermatology?124, 6 (1988), 869–871.
    61. Lou Gevaux, Cyprien Adnet, Pierre S?roul, Raphael Clerc, Alain Tr?meau, Jean Luc Perrot, and Mathieu H?bert. 2019. Three-dimensional maps of human skin properties on full face with shadows using 3-D hyperspectral imaging.?Journal of biomedical optics?24, 6 (2019), 066002.
    62. Lou Gevaux, Jordan Gierschendorf, Juliette Rengot, Marie Cherel, Pierre S?roul, Alex Nkengne, Julie Robic, Alain Tr?meau, and Mathieu H?bert. 2021. Real-time skin chromophore estimation from hyperspectral images using a neural network.?Skin Research and Technology?27, 2 (2021), 163–177.
    63. Yuliya Gitlina, Giuseppe Claudio Guarnera, Daljit Singh Dhillon, Jan Hansen, Alexandros Lattas, Dinesh Pai, and Abhijeet Ghosh. 2020. Practical Measurement and Reconstruction of Spectral Skin Reflectance. In?Computer Graphics Forum, Vol. 39. Wiley Online Library, 75–89.
    64. Louis G Henyey and Jesse L Greenstein. 1941. Diffuse radiation in the galaxy.?The Astrophysical Journal?93 (1941), 70–83.
    65. Jose A. Iglesias-Guitian, Carlos Aliaga, Adrian Jarabo, and Diego Gutierrez. 2015. A Biophysically-Based Model of the Optical Properties of Skin Aging.?Computer Graphics Forum (EUROGRAPHICS 2015)?34, 2 (2015).
    66. Steven L Jacques. 2013. Optical properties of biological tissues: a review.?Physics in Medicine & Biology?58, 11 (2013), R37.
    67. Steven L Jacques and Daniel J McAuliffe. 1991. The melanosome: threshold temperature for explosive vaporization and internal absorption coefficient during pulsed laser irradiation.?Photochemistry and photobiology?53, 6 (1991), 769–775.
    68. Wenzel Jakob and Johannes Hanika. 2019. A Low-Dimensional Function Space for Efficient Spectral Upsampling.?Computer Graphics Forum (Proceedings of Eurographics)?38, 2 (March 2019).
    69. Adrian Jarabo, Carlos Aliaga, and Diego Gutierrez. 2018. A radiative transfer framework for spatially-correlated materials.?ACM Transactions on Graphics (TOG) 37, 4 (2018), 1–13.
    70. Nirmal Keshava and John F Mustard. 2002. Spectral unmixing.?IEEE signal processing magazine?19, 1 (2002), 44–57.
    71. Aravind Krishnaswamy and Gladimir VG Baranoski. 2004. A biophysically-based spectral model of light interaction with human skin. In?Computer Graphics Forum, Vol. 23. Wiley Online Library, 331–340.
    72. Tom Lister, Philip A. Wright, and Paul H. Chappell. 2012. Optical properties of human skin.?Journal of Biomedical Optics?17, 9 (2012), 1 — 15.
    73. Ian Mallett and Cem Yuksel. 2019. Spectral Primary Decomposition for Rendering with RGB Reflectance. In?Eurographics Symposium on Rendering (EGSR 2019)?(Strasbourg, France). The Eurographics Association.
    74. Igor V Meglinski and Stephen J Matcher. 2002. Quantitative assessment of skin layers absorption and skin reflectance spectra simulation in the visible and near-infrared spectral regions.?Physiological measurement?23, 4 (2002), 741.
    75. Scott Prahl and Steven L Jacques. [n.d.]. Tabulated Molar Extinction Coefficient for Hemoglobin in Water. https://omlc.org/spectra/hemoglobin/summary.html.
    76. Iyad Salam Saidi. 1992.?Transcutaneous optical measurement of hyperbilirubinemia in neonates.?Ph.D. Dissertation.
    77. Tadeusz Sarna and Harold A Swartz. 2006. The physical properties of melanins.?The pigmentary system: physiology and pathophysiology?(2006), 311–341.
    78. MJC Van Gemert, Steven L Jacques, HJCM Sterenborg, and WM Star. 1989. Skin optics.?IEEE Transactions on biomedical engineering?36, 12 (1989), 1146–1154.
    79. Bruce Walter, Stephen R. Marschner, Hongsong Li, and Kenneth E. Torrance. 2007. Micro-facet Models for Refraction through Rough Surfaces. In?Proceedings of the 18th Eurographics Conference on Rendering Techniques?(Grenoble, France)?(EGSR’07).?Eurographics Association, Goslar, DEU, 195–206.
    80. Magnus Wrenninge, Ryusuke Villemin, and Christophe Hery. 2017.?Path traced subsurface scattering using anisotropic phase functions and non-exponential free flights.?Technical Report. Technical Memo.
    81. Evgeny Zherebtsov, Viktor Dremin, Alexey Popov, Alexander Doronin, Daria Kurakina, Mikhail Kirillin, Igor Meglinski, and Alexander Bykov. 2019. Hyperspectral imaging of human skin aided by artificial neural networks.?Biomedical optics express?10, 7 (2019), 3545–3559.
    82. George Zonios, Julie Bykowski, and Nikiforos Kollias. 2001. Skin melanin, hemoglobin, and light scattering properties can be quantitatively assessed in vivo using diffuse reflectance spectroscopy.?Journal of Investigative Dermatology?117, 6 (2001), 1452–1457.
    83. Benedikt Bitterli, Chris Wyman, Matt Pharr, Peter Shirley, Aaron Lefohn, and Wojciech Jarosz. 2020. Spatiotemporal reservoir resampling for real-time ray tracing with dynamic direct lighting.?ACM Transactions on Graphics (Proceedings of SIGGRAPH)?39, 4 (July 2020).?
    84. Guillaume Boiss?. 2021.?World-space spatiotemporal reservoir reuse for ray-traced Global illumination.?Association for Computing Machinery, New York, NY, USA.?
    85. Jakub Boksansky, Paula Jukarainen, and Chris Wyman. 2021.?Rendering Many Lights with Grid-Based Reservoirs.?Apress, Berkeley, CA, 351–365.?
    86. Daqi Lin, Markus Kettunen, Benedikt Bitterli, Jacopo Pantaleoni, Cem Yuksel, and Chris Wyman. 2022. Generalized Resampled Importance Sampling: Foundations of ReSTIR.?ACM Transactions on Graphics (Proceedings of SIGGRAPH 2022)?41, 4, Article 75 (07 2022), 23 pages.?
    87. Daqi Lin, Chris Wyman, and Cem Yuksel. 2021. Fast Volume Rendering with Spatiotemporal Reservoir Resampling.?ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia 2021)?40, 6, Article 279 (dec 2021), 18 pages.?
    88. Y. Ouyang, S. Liu, M. Kettunen, M. Pharr, and Jacopo Pantaleoni. 2021. ReSTIR GI: Path Resampling for Real-Time Path Tracing.?Computer Graphics Forum?40 (12 2021), 17–29.?
    89. Justin Talbot, David Cline, and Parris Egbert. 2005. Importance Resampling for Global Illumination.?Eurographics Symposium on Rendering, 139–146.?
    90. M. van de Ruit and E. Eisemann. 2021. A multi-pass method for accelerated spectral sampling.?Computer Graphics Forum?40, 7 (2021), 141–148. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/cgf.14408?
    91. Rex West, Iliyan Georgiev, Adrien Gruson, and Toshiya Hachisuka. 2020. Continuous Multiple Importance Sampling.?ACM Trans. Graph.?39, 4, Article 136 (July 2020), 12 pages.?
    92. A Wilkie, S Nawaz, Marc Droske, A Weidlich, and J Hanika. 2014. Hero Wavelength Spectral Sampling.?Computer Graphics Forum?33 (07 2014).

ACM Digital Library Publication:


Overview Page:


Submit a story:

If you would like to submit a story about this presentation, please contact us: historyarchives@siggraph.org