“Machine Learning Meets Lighting: Using Depth Estimation To Build The Light Rigs” by Shlyaev
Conference:
Type(s):
Title:
- Machine Learning Meets Lighting: Using Depth Estimation To Build The Light Rigs
Session/Category Title:
- ML in Production
Presenter(s)/Author(s):
Moderator(s):
Abstract:
This paper describes the techniques we use to build the complex light rigs in the Sony Pictures Imageworks lighting pipeline. Typically we receive the panoramic HDRI from the set and need to make a light rig from it. Building a light rig has several steps: extracting area lights from HDRI, placing them in a 3D scene, aligning lights with Lidar from set. We describe how this process can be sped up and automated using modern machine learning and computer vision techniques.
References:
[1] Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller. 2023. Zoedepth: Zero-shot transfer by combining relative and metric depth. arXiv preprint arXiv:2302.12288 (2023).
[2] G. Bradski. 2000. The OpenCV Library. Dr. Dobb’s Journal of Software Tools (2000).
[3] Academy Software Foundation. 2025. OpenImageIO. https://openimageio.org. Version 3.0.
[4] W. Kabsch. 1978. A solution for the best rotation to relate two sets of vectors. Acta Crystallographica Section A: Crystal Physics, Diffraction, Theoretical and General Crystallography 34 (1978), 827–828.
[5] Zhenyu Li, Shariq Farooq Bhat, and Peter Wonka. 2023. PatchFusion: An End-to-End Tile-Based Framework for High-Resolution Monocular Metric Depth Estimation. arxiv:2312.02284 [cs.CV] https://arxiv.org/abs/2312.02284
[6] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. 2022. Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer. IEEE Transactions on Pattern Analysis and Machine Intelligence 44, 3 (2022).
[7] S. Umeyama. 1991. Least-squares estimation of transformation parameters between two point patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence 13, 4 (April 1991), 376–380.
