“Practical aspects of spectral data in digital content production” Chaired by

  • ©Andrea Weidlich, Chloe LeGendre, Carlos Aliaga, Christophe Hery, Jean-Marie Aubry, Jiří Vorba, Daniele Siragusano, and Richard Kirk

Conference:


Type:


Entry Number: 11

Title:

    Practical aspects of spectral data in digital content production

Presenter(s)/Author(s):


Abstract:


    Compared to path tracing, spectral rendering is still often considered to be a niche application used mainly to produce optical wave effects like dispersion or diffraction. And while over the last years more and more people started exploring the potential of spectral image synthesis, it is still widely assumed to be only of importance in high-quality offline applications associated with long render times and high visual fidelity.

    While it is certainly true that describing light interactions in a spectral way is a necessity for predictive rendering, its true potential goes far beyond that. Used correctly, not only will it guarantee colour fidelity, but it will also simplify workflows for all sorts of applications.

    Wētā Digital’s renderer Manuka showed that there is a place for a spectral renderer in a production environment and how workflows can be simplified if the whole pipeline adapts. Picking up from the course last year, we want to continue the discussion we started as we firmly believe that spectral data is the future in content production. The authors feel enthusiastic about more people being aware of the advantages that spectral rendering and spectral workflows bring and share the knowledge we gained over many years. The novel workflows emerged during the adaptation of spectral techniques at a number of large companies are introduced to a wide audience including technical directors, artists and researchers. However, while last year’s course concentrated primarily on the algorithmic sides of spectral image synthesis, this year we want to focus on the practical aspects.

    We will draw examples from virtual production, digital humans over spectral noise reduction to image grading, therefore showing the usage of spectral data enhancing each and every single part of the image pipeline.

References:


    1. Catherine Cooksey, David Allen, and Benjamin Tsai. 2017. Reference Data Set of Human Skin Reflectance. 
    2. Weta Digital. 2022. Physlight. Physlight
    3. R Donaldson. 1954. Spectrophotometry of fluorescent pigments. British Journal of Applied Physics 5, 6 (jun 1954), 210–214. 
    4. Luca Fascione, Johannes Hanika, Mark Leone, Marc Droske, Jorge Schwarzhaupt, Tomas Davidovic, Andrea Weidlich, and Johannes Meng. 2018. Manuka: A Batch-Shading Architecture for Spectral Path Tracing in Movie Production. ACM Transactions on Graphics 37 (08 2018), 1–18. 
    5. Alban Fichet, Romain Pacanowski, and Alexander Wilkie. 2021. An OpenEXR Layout for Spectral images. Journal of Computer Graphics Techniques (JCGT) 10, 3 (29 September 2021), 1–18. http://jcgt.org/published/0010/03/01/
    6. Qingqin Hua, Alban Fichet, and Alexander Wilkie. 2021. A Compact Representation for Fluorescent Spectral Data. In Eurographics Symposium on Rendering – DL-only Track, Adrien Bousseau and Morgan McGuire (Eds.). The Eurographics Association. 
    7. Longqian Huang, Ruichen Luo, Xu Liu, and Xiang Hao. 2022. Spectral imaging with deep learning. Light: Science & Applications 11 (03 2022), 61. 
    8. Wenzel Jakob and Johannes Hanika. 2019. A Low-Dimensional Function Space for Efficient Spectral Upsampling. Computer Graphics Forum (Proceedings of Eurographics) 38, 2 (March 2019).
    9. Alisa Jung, A. Wilkie, J. Hanika, W. Jakob, and C. Dachsbacher. 2019. Wide Gamut Spectral Upsampling with Fluorescence. Computer Graphics Forum 38 (07 2019), 87–96. 
    10. Lars König, Alisa Jung, and Carsten Dachsbacher. 2020. Improving Spectral Upsampling with Fluorescence. In Workshop on Material Appearance Modeling, Reinhard Klein and Holly Rushmeier (Eds.). The Eurographics Association. 
    11. Tom Lister, Philip A. Wright, and Paul H. Chappell. 2012. Optical properties of human skin. Journal of Biomedical Optics 17, 9 (2012), 1 — 15. 
    12. Ian Mallett and Cem Yuksel. 2019. Spectral Primary Decomposition for Rendering with sRGB Reflectance. In Eurographics Symposium on Rendering – DL-only and Industry Track, Tamy Boubekeur and Pradeep Sen (Eds.). The Eurographics Association. 
    13. Johannes Meng, Florian Simon, Johannes Hanika, and Carsten Dachsbacher. 2015. Physically meaningful rendering using tristimulus colours. In Computer Graphics Forum, Vol. 34. Wiley Online Library, 31–40.
    14. Arash Mirhashemi. 2018. Introducing spectral moment features in analyzing the SpecTex hyperspectral texture database. Machine Vision and Applications 29 (04 2018). 
    15. Hisanari Otsu, Masafumi Yamamoto, and Toshiya Hachisuka. 2018. Reproducing spectral reflectances from tristimulus colours. In Computer Graphics Forum, Vol. 37. Wiley Online Library, 370–381.
    16. Mark S. Peercy. 1993. Linear Color Representations for Full Speed Spectral Rendering. In Proceedings of the 20th Annual Conference on Computer Graphics and Interactive Techniques (Anaheim, CA) (SIGGRAPH ’93). Association for Computing Machinery, New York, NY, USA, 191–198. 
    17. Scyllarus. 2016. HSZ Data Format. https://scyllarus.data61.csiro.au/data/hsz-data-format/
    18. Brian Smits. 1999. An RGB-to-spectrum conversion for reflectances. Journal of Graphics Tools 4, 4 (1999), 11–22.
    19. Yinlong Sun, M.S. Drew, and F.D. Fracchia. 1999. Representing spectral functions by a composite model of smooth and spiky components for efficient full-spectrum photorealism. In Proceedings Workshop on Photometric Modeling for Computer Vision and Graphics (Cat. No.PR00271). 4–11. 
    20. Lucia Tódová, Alexander Wilkie, and Luca Fascione. 2021. Moment-based Constrained Spectral Uplifting. In Eurographics Symposium on Rendering – DL-only Track, Adrien Bousseau and Morgan McGuire (Eds.). The Eurographics Association. 
    21. Mark van de Ruit and Elmar Eisemann. 2021. A multi-pass method for accelerated spectral sampling. Computer Graphics Forum 40 (2021).
    22. Andrea Weidlich, Alex Forsythe, Scott Dyer, Thomas Mansencal, Johannes Hanika, Alexander Wilkie, Luke Emrose, and Anders Langlands. 2021. Spectral Imaging in Production: Course Notes Siggraph 2021. In ACM SIGGRAPH 2021 Courses (Virtual Event, USA) (SIGGRAPH ’21). Association for Computing Machinery, New York, NY, USA, Article 14, 90 pages. 
    23. Rex West, Iliyan Georgiev, Adrien Gruson, and Toshiya Hachisuka. 2020. Continuous Multiple Importance Sampling. ACM Trans. Graph. 39, 4, Article 136 (jul 2020), 12 pages. 
    24. A. Wilkie, S. Nawaz, M. Droske, A. Weidlich, and J. Hanika. 2014. Hero Wavelength Spectral Sampling. In Proceedings of the 25th Eurographics Symposium on Rendering (Lyon, France) (EGSR ’14). Eurographics Association, Goslar, DEU, 123–131.
    25. Alexander Wilkie, Robert Tobler, and Werner Purgathofer. 2000. Raytracing of Dispersion Effects in Transparent Materials.
    26. Chris Wyman, Peter-Pike Sloan, and Peter Shirley. 2013. Simple Analytic Approximations to the CIE XYZ Color Matching Functions. Journal of Computer Graphics Techniques (JCGT) 2, 2 (12 July 2013), 1–11. http://jcgt.org/published/0002/02/01/
    27. X-Rite. 2022. AxF Data Format. https://www.xrite.com/appearance-exchange-format-axf
    28. [n.d.]. In-camera VFX Camera Color Calibration. https://docs.unrealengine.com/4.27/en-US/WorkingWithMedia/IntegratingMedia/InCameraVFX/InCameraVFXCameraCalibration/
    29. Carlos F Borges. 1991. Trichromatic approximation for computer graphics illumination models. In Proceedings of the 18th annual conference on Computer graphics and interactive techniques (SIGGRAPH ’91). 101–104.
    30. Paul Debevec. 1998. Rendering Synthetic Objects into Real Scenes: Bridging Traditional and Image-Based Graphics with Global Illumination and High Dynamic Range Photography. In Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’98). Association for Computing Machinery, New York, NY, USA, 189–198. 
    31. Paul Debevec, Andreas Wenger, Chris Tchou, Andrew Gardner, Jamie Waese, and Tim Hawkins. 2002. A Lighting Reproduction Approach to Live-Action Compositing. ACM Trans. Graph. 21, 3 (Jul 2002), 547–556. 
    32. Jonathan Erland. 2011. Chromatic Chaos: Implications of Newly Introduced Forms of Stage-light. National Association of Broadcasters (NAB) Conference.
    33. Nahum Gat and CA Torrance. 1998. Real-time multi-and hyper-spectral imaging for remote sensing and machine vision: an overview. In Proc. 1998 ASAE Annual International Mtg. Citeseer.
    34. Pierre-Loïc Hamon, James Harmer, Stuart Penn, and Nicolas Scapel. 2014. Gravity: Motion control and face integration. In ACM SIGGRAPH 2014 Talks. 1–1.
    35. Jay Holben. 2020. The Mandalorian: This Is the Way. https://ascmag.com/articles/the-mandalorian
    36. Jack Holm, Tom Maier, Paul Debevec, Chloe LeGendre, Joshua Pines, Jonathan Erland, George Joblove, Scott Dyer, Blake Sloan, Joe di Gennaro, et al. 2016. A cinematographic spectral similarity index. In SMPTE 2016 Annual Technical Conference and Exhibition. SMPTE, 1–36.
    37. Jun Jiang, Dengyu Liu, Jinwei Gu, and Sabine Süsstrunk. 2013. What is the space of spectral sensitivity functions for digital color cameras?. In 2013 IEEE Workshop on Applications of Computer Vision (WACV). IEEE, 168–179.
    38. Noah Kadner. 2021a. 1899 Wraps Innovative Virtual Production. https://ascmag.com/articles/1899-wraps-virtual-production
    39. Noah Kadner. 2021b. Color Fidelity in LED Volumes. https://ascmag.com/articles/color-fidelity-in-led-volumes
    40. Noah Kadner. 2021c. On The Walls: Virtual Production for Series Shooting. https://ascmag.com/articles/on-the-walls
    41. Chloe LeGendre, Lukas Lepicovsky, and Paul Debevec. 2022. Jointly Optimizing Color Rendition and In-Camera Backgrounds in an RGB Virtual Production Stage. (arXiv.org) (2022).
    42. Chloe LeGendre, Xueming Yu, and Paul Debevec. 2017. Optimal LED selection for multi-spectral lighting reproduction. Electronic Imaging 2017, 8 (2017), 25–32.
    43. Chloe LeGendre, Xueming Yu, Dai Liu, Jay Busch, Andrew Jones, Sumanta Pattanaik, and Paul Debevec. 2016. Practical multispectral lighting reproduction. ACM Transactions on Graphics (TOG) 35, 4 (2016), 1–11.
    44. Calvin S McCamy, Harold Marcus, James G Davidson, et al. 1976. A color-rendition chart. J. App. Photog. Eng 2, 3 (1976), 95–99.
    45. Muhammad Uzair, Arif Mahmood, Faisal Shafait, Christian Nansen, and Ajmal Mian. 2015. Is spectral reflectance of the face a reliable biometric? Optics express 23, 12 (2015), 15160–15173.
    46. Andrea Weidlich, Alex Forsythe, Scott Dyer, Thomas Mansencal, Johannes Hanika, Alexander Wilkie, Luke Emrose, and Anders Langlands. 2021. Spectral imaging in production: course notes Siggraph 2021. In ACM SIGGRAPH 2021 Courses. 1–90.
    47. Andreas Wenger, Tim Hawkins, and Paul Debevec. 2003. Optimizing color matching in a lighting reproduction system for complex subject and illuminant spectra. In Rendering Techniques. Citeseer, 249–259.
    48. Carlos Aliaga, Christophe Hery, and Mengqi Xia. 2022. Estimation of Spectral Biophysical Skin Properties from Captured RGB Albedo.
    49. Sarah Alotaibi and William AP Smith. 2017. A biophysical 3D morphable model of face appearance. In Proceedings of the IEEE International Conference on Computer Vision Workshops. 824–832.
    50. Gladimir VG Baranoski and Aravind Krishnaswamy. 2010. Light and skin interactions: simulations for computer graphics applications. Morgan Kaufmann.
    51. Alexey N Bashkatov, Elina A Genina, and Valery V Tuchin. 2011. Optical properties of skin, subcutaneous, and muscle tissues: a review. Journal of Innovative Optical Health Sciences 4, 01 (2011), 9–38.
    52. Benedikt Bitterli, Srinath Ravichandran, Thomas Müller, Magnus Wrenninge, Jan Novák, Steve Marschner, and Wojciech Jarosz. 2018. A Radiative Transfer Framework for Non-Exponential Media. ACM Trans. Graph. 37, 6, Article 225 (Dec. 2018), 17 pages.
    53. Tenn F Chen, Gladimir VG Baranoski, Bradley W Kimmel, and Erik Miranda. 2015. Hyper-spectral modeling of skin appearance. ACM Transactions on Graphics (TOG) 34, 3 (2015), 1–14.
    54. Blender Online Community. 2020. Blender: a 3D modelling and rendering package. (2020). http://www.blender.org
    55. Carolin Czekalla, Karl Heinz Schönborn, Jürgen Lademann, and Martina C Meinke. 2019. Noninvasive determination of epidermal and stratum corneum thickness in vivo using two-photon microscopy and optical coherence tomography: Impact of body area, age, and gender. Skin pharmacology and physiology 32, 3 (2019), 142–150.
    56. Philip E Dennison, Kerry Q Halligan, and Dar A Roberts. 2004. A comparison of error metrics and constraints for multiple endmember spectral mixture analysis and spectral angle mapper. Remote Sensing of Environment 93, 3 (2004), 359–367.
    57. Craig Donner and Henrik Wann Jensen. 2006. A Spectral BSSRDF for Shading Human Skin. Rendering techniques 2006 (2006), 409–418.
    58. Eugene d’ Eon. 2019. A Reciprocal Formulation of Nonexponential Radiative Transfer. 2: Monte Carlo Estimation and Diffusion Approximation. Journal of Computational and Theoretical Transport 48, 6 (Sep 2019), 201–262.
    59. Eugene d’Eon. 2021. A Hitchhiker’s Guide to Multiple Scattering. (2021).
    60. Thomas B Fitzpatrick. 1988. The validity and practicality of sun-reactive skin types I through VI. Archives of dermatology 124, 6 (1988), 869–871.
    61. Lou Gevaux, Cyprien Adnet, Pierre Séroul, Raphael Clerc, Alain Trémeau, Jean Luc Perrot, and Mathieu Hébert. 2019. Three-dimensional maps of human skin properties on full face with shadows using 3-D hyperspectral imaging. Journal of biomedical optics 24, 6 (2019), 066002.
    62. Lou Gevaux, Jordan Gierschendorf, Juliette Rengot, Marie Cherel, Pierre Séroul, Alex Nkengne, Julie Robic, Alain Trémeau, and Mathieu Hébert. 2021. Real-time skin chromophore estimation from hyperspectral images using a neural network. Skin Research and Technology 27, 2 (2021), 163–177.
    63. Yuliya Gitlina, Giuseppe Claudio Guarnera, Daljit Singh Dhillon, Jan Hansen, Alexandros Lattas, Dinesh Pai, and Abhijeet Ghosh. 2020. Practical Measurement and Reconstruction of Spectral Skin Reflectance. In Computer Graphics Forum, Vol. 39. Wiley Online Library, 75–89.
    64. Louis G Henyey and Jesse L Greenstein. 1941. Diffuse radiation in the galaxy. The Astrophysical Journal 93 (1941), 70–83.
    65. Jose A. Iglesias-Guitian, Carlos Aliaga, Adrian Jarabo, and Diego Gutierrez. 2015. A Biophysically-Based Model of the Optical Properties of Skin Aging. Computer Graphics Forum (EUROGRAPHICS 2015) 34, 2 (2015).
    66. Steven L Jacques. 2013. Optical properties of biological tissues: a review. Physics in Medicine & Biology 58, 11 (2013), R37.
    67. Steven L Jacques and Daniel J McAuliffe. 1991. The melanosome: threshold temperature for explosive vaporization and internal absorption coefficient during pulsed laser irradiation. Photochemistry and photobiology 53, 6 (1991), 769–775.
    68. Wenzel Jakob and Johannes Hanika. 2019. A Low-Dimensional Function Space for Efficient Spectral Upsampling. Computer Graphics Forum (Proceedings of Eurographics) 38, 2 (March 2019).
    69. Adrian Jarabo, Carlos Aliaga, and Diego Gutierrez. 2018. A radiative transfer framework for spatially-correlated materials. ACM Transactions on Graphics (TOG) 37, 4 (2018), 1–13.
    70. Nirmal Keshava and John F Mustard. 2002. Spectral unmixing. IEEE signal processing magazine 19, 1 (2002), 44–57.
    71. Aravind Krishnaswamy and Gladimir VG Baranoski. 2004. A biophysically-based spectral model of light interaction with human skin. In Computer Graphics Forum, Vol. 23. Wiley Online Library, 331–340.
    72. Tom Lister, Philip A. Wright, and Paul H. Chappell. 2012. Optical properties of human skin. Journal of Biomedical Optics 17, 9 (2012), 1 — 15.
    73. Ian Mallett and Cem Yuksel. 2019. Spectral Primary Decomposition for Rendering with RGB Reflectance. In Eurographics Symposium on Rendering (EGSR 2019) (Strasbourg, France). The Eurographics Association.
    74. Igor V Meglinski and Stephen J Matcher. 2002. Quantitative assessment of skin layers absorption and skin reflectance spectra simulation in the visible and near-infrared spectral regions. Physiological measurement 23, 4 (2002), 741.
    75. Scott Prahl and Steven L Jacques. [n.d.]. Tabulated Molar Extinction Coefficient for Hemoglobin in Water. https://omlc.org/spectra/hemoglobin/summary.html.
    76. Iyad Salam Saidi. 1992. Transcutaneous optical measurement of hyperbilirubinemia in neonates. Ph.D. Dissertation.
    77. Tadeusz Sarna and Harold A Swartz. 2006. The physical properties of melanins. The pigmentary system: physiology and pathophysiology (2006), 311–341.
    78. MJC Van Gemert, Steven L Jacques, HJCM Sterenborg, and WM Star. 1989. Skin optics. IEEE Transactions on biomedical engineering 36, 12 (1989), 1146–1154.
    79. Bruce Walter, Stephen R. Marschner, Hongsong Li, and Kenneth E. Torrance. 2007. Micro-facet Models for Refraction through Rough Surfaces. In Proceedings of the 18th Eurographics Conference on Rendering Techniques (Grenoble, France) (EGSR’07). Eurographics Association, Goslar, DEU, 195–206.
    80. Magnus Wrenninge, Ryusuke Villemin, and Christophe Hery. 2017. Path traced subsurface scattering using anisotropic phase functions and non-exponential free flights. Technical Report. Technical Memo.
    81. Evgeny Zherebtsov, Viktor Dremin, Alexey Popov, Alexander Doronin, Daria Kurakina, Mikhail Kirillin, Igor Meglinski, and Alexander Bykov. 2019. Hyperspectral imaging of human skin aided by artificial neural networks. Biomedical optics express 10, 7 (2019), 3545–3559.
    82. George Zonios, Julie Bykowski, and Nikiforos Kollias. 2001. Skin melanin, hemoglobin, and light scattering properties can be quantitatively assessed in vivo using diffuse reflectance spectroscopy. Journal of Investigative Dermatology 117, 6 (2001), 1452–1457.
    83. Benedikt Bitterli, Chris Wyman, Matt Pharr, Peter Shirley, Aaron Lefohn, and Wojciech Jarosz. 2020. Spatiotemporal reservoir resampling for real-time ray tracing with dynamic direct lighting. ACM Transactions on Graphics (Proceedings of SIGGRAPH) 39, 4 (July 2020). 
    84. Guillaume Boissé. 2021. World-space spatiotemporal reservoir reuse for ray-traced Global illumination. Association for Computing Machinery, New York, NY, USA. 
    85. Jakub Boksansky, Paula Jukarainen, and Chris Wyman. 2021. Rendering Many Lights with Grid-Based Reservoirs. Apress, Berkeley, CA, 351–365. 
    86. Daqi Lin, Markus Kettunen, Benedikt Bitterli, Jacopo Pantaleoni, Cem Yuksel, and Chris Wyman. 2022. Generalized Resampled Importance Sampling: Foundations of ReSTIR. ACM Transactions on Graphics (Proceedings of SIGGRAPH 2022) 41, 4, Article 75 (07 2022), 23 pages. 
    87. Daqi Lin, Chris Wyman, and Cem Yuksel. 2021. Fast Volume Rendering with Spatiotemporal Reservoir Resampling. ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia 2021) 40, 6, Article 279 (dec 2021), 18 pages. 
    88. Y. Ouyang, S. Liu, M. Kettunen, M. Pharr, and Jacopo Pantaleoni. 2021. ReSTIR GI: Path Resampling for Real-Time Path Tracing. Computer Graphics Forum 40 (12 2021), 17–29. 
    89. Justin Talbot, David Cline, and Parris Egbert. 2005. Importance Resampling for Global Illumination. Eurographics Symposium on Rendering, 139–146. 
    90. M. van de Ruit and E. Eisemann. 2021. A multi-pass method for accelerated spectral sampling. Computer Graphics Forum 40, 7 (2021), 141–148. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/cgf.14408 
    91. Rex West, Iliyan Georgiev, Adrien Gruson, and Toshiya Hachisuka. 2020. Continuous Multiple Importance Sampling. ACM Trans. Graph. 39, 4, Article 136 (July 2020), 12 pages. 
    92. A Wilkie, S Nawaz, Marc Droske, A Weidlich, and J Hanika. 2014. Hero Wavelength Spectral Sampling. Computer Graphics Forum 33 (07 2014).

ACM Digital Library Publication:


Overview Page: