“EyeIR: Single Eye Image Inverse Rendering in the Wild”
Conference:
Type(s):
Title:
- EyeIR: Single Eye Image Inverse Rendering in the Wild
Presenter(s)/Author(s):
Abstract:
We propose a method to decompose a single eye image into albedo, shading, specular, normal, and illumination component. We create a diverse synthetic eye dataset and design a synthetic-to-real adaptation framework for self-supervised learning on real images. We also design a method which specifically addresses ambiguities caused by corneal specularity.
References:
[1]
Thiemo Alldieck, Mihai Zanfir, and Cristian Sminchisescu. 2022. Photorealistic monocular 3d reconstruction of humans wearing clothing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1506?1515.
[2]
Jonathan T Barron and Jitendra Malik. 2014. Shape, illumination, and reflectance from shading. IEEE transactions on pattern analysis and machine intelligence 37, 8 (2014), 1670?1687.
[3]
Anil S Baslamisli, Partha Das, Hoang-An Le, Sezer Karaoglu, and Theo Gevers. 2021. Shadingnet: Image intrinsics by fine-grained shading decomposition. International Journal of Computer Vision 129, 8 (2021), 2445?2473.
[4]
Anil S Baslamisli, Hoang-An Le, and Theo Gevers. 2018. CNN based learning using reflection and retinex models for intrinsic image decomposition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 6674?6683.
[5]
Pascal B?rard, Derek Bradley, Markus Gross, and Thabo Beeler. 2016. Lightweight eye capture using a parametric model. ACM Transactions on Graphics (TOG) 35, 4 (2016), 1?12.
[6]
Pascal B?rard, Derek Bradley, Maurizio Nitti, Thabo Beeler, and Markus H Gross. 2014. High-quality capture of eyes.ACM Trans. Graph. 33, 6 (2014), 223?1.
[7]
Wenzheng Chen, Huan Ling, Jun Gao, Edward Smith, Jaakko Lehtinen, Alec Jacobson, and Sanja Fidler. 2019. Learning to predict 3d objects with an interpolation-based differentiable renderer. Advances in neural information processing systems 32 (2019).
[8]
Wenzheng Chen, Joey Litalien, Jun Gao, Zian Wang, Clement Fuji Tsang, Sameh Khamis, Or Litany, and Sanja Fidler. 2021. DIB-R++: learning to predict lighting and material with a hybrid differentiable renderer. Advances in Neural Information Processing Systems 34 (2021), 22834?22848.
[9]
Partha Das, Sezer Karaoglu, and Theo Gevers. 2022. PIE-Net: Photometric Invariant Edge Guided Network for Intrinsic Image Decomposition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 19790?19799.
[10]
Xuanyi Dong, Shoou-I Yu, Xinshuo Weng, Shih-En Wei, Yi Yang, and Yaser Sheikh. 2018. Supervision-by-registration: An unsupervised approach to improve the precision of facial landmark detectors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 360?368.
[11]
Gang Fu, Qing Zhang, Lei Zhu, Ping Li, and Chunxia Xiao. 2021. A multi-task network for joint specular highlight detection and removal. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 7752?7761.
[12]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770?778.
[13]
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016. Perceptual losses for real-time style transfer and super-resolution. In Computer Vision?ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14. Springer, 694?711.
[14]
Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. 2020. MaskGAN: Towards Diverse and Interactive Facial Image Manipulation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
[15]
Chen Li, Stephen Lin, Kun Zhou, and Katsushi Ikeuchi. 2017. Specular highlight removal in facial images. In Proceedings of the ieee conference on computer vision and pattern recognition. 3107?3116.
[16]
Chen Li, Kun Zhou, and Stephen Lin. 2014. Intrinsic face image decomposition with human face priors. In Computer Vision?ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. Springer, 218?233.
[17]
Gengyan Li, Abhimitra Meka, Franziska Mueller, Marcel C Buehler, Otmar Hilliges, and Thabo Beeler. 2022. EyeNeRF: a hybrid representation for photorealistic synthesis, animation and relighting of human eyes. ACM Transactions on Graphics (TOG) 41, 4 (2022), 1?16.
[18]
Zhengqi Li and Noah Snavely. 2018. Cgintrinsics: Better intrinsic image decomposition through physically-based rendering. In Proceedings of the European conference on computer vision (ECCV). 371?387.
[19]
Zhengqin Li, Zexiang Xu, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker. 2018. Learning to reconstruct shape and spatially-varying reflectance from a single image. ACM Trans. Graph. 37, 6, Article 269 (dec 2018), 11 pages. https://doi.org/10.1145/3272127.3275055
[20]
Guilin Liu, Duygu Ceylan, Ersin Yumer, Jimei Yang, and Jyh-Ming Lien. 2017. Material editing using a physically based rendering network. In Proceedings of the IEEE International Conference on Computer Vision. 2261?2269.
[21]
Yunfei Liu, Yu Li, Shaodi You, and Feng Lu. 2020. Unsupervised learning for intrinsic image decomposition from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 3248?3257.
[22]
Jundan Luo, Zhaoyang Huang, Yijin Li, Xiaowei Zhou, Guofeng Zhang, and Hujun Bao. 2020. NIID-Net: adapting surface normal knowledge for intrinsic image decomposition in indoor scenes. IEEE Transactions on Visualization and Computer Graphics 26, 12 (2020), 3434?3445.
[23]
Jiangjing Lv, Xiaohu Shao, Junliang Xing, Cheng Cheng, and Xi Zhou. 2017. A deep regression architecture with two-stage re-initialization for high performance facial landmark detection. In Proceedings of the IEEE conference on computer vision and pattern recognition. 3317?3326.
[24]
Takuya Narihira, Michael Maire, and Stella X Yu. 2015. Direct intrinsics: Learning albedo-shading decomposition by convolutional regression. In Proceedings of the IEEE international conference on computer vision. 2992?2992.
[25]
Ko Nishino and S.K. Nayar. 2004. The world in an eye [eye image interpretation]. In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004., Vol. 1. I?I. https://doi.org/10.1109/CVPR.2004.1315066
[26]
Xingang Pan, Ayush Tewari, Lingjie Liu, and Christian Theobalt. 2022. GAN2X: Non-Lambertian Inverse Rendering of Image GANs. In International Conference on 3D Vision. IEEE.
[27]
Stylianos Ploumpis, Evangelos Ververas, Eimear O?Sullivan, Stylianos Moschoglou, Haoyang Wang, Nick Pears, William AP Smith, Baris Gecer, and Stefanos Zafeiriou. 2020. Towards a complete 3D morphable model of the human head. IEEE transactions on pattern analysis and machine intelligence 43, 11 (2020), 4142?4160.
[28]
Ravi Ramamoorthi and Pat Hanrahan. 2001. An efficient representation for irradiance environment maps. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques. 497?500.
[29]
Elad Richardson, Matan Sela, Roy Or-El, and Ron Kimmel. 2017. Learning detailed face reconstruction from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1259?1268.
[30]
Leonid I Rudin, Stanley Osher, and Emad Fatemi. 1992. Nonlinear total variation based noise removal algorithms. Physica D: nonlinear phenomena 60, 1-4 (1992), 259?268.
[31]
Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. 2019. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In Proceedings of the IEEE/CVF international conference on computer vision. 2304?2314.
[32]
Shen Sang and Manmohan Chandraker. 2020. Single-shot neural relighting and svbrdf estimation. In Computer Vision?ECCV 2020: 16th European Conference, Glasgow, UK, August 23?28, 2020, Proceedings, Part XIX 16. Springer, 85?101.
[33]
Soumyadip Sengupta, Jinwei Gu, Kihwan Kim, Guilin Liu, David W Jacobs, and Jan Kautz. 2019. Neural inverse rendering of an indoor scene from a single image. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 8598?8607.
[34]
Soumyadip Sengupta, Angjoo Kanazawa, Carlos D Castillo, and David W Jacobs. 2018. Sfsnet: Learning shape, reflectance and illuminance of facesin the wild?. In Proceedings of the IEEE conference on computer vision and pattern recognition. 6296?6305.
[35]
Steven A Shafer. 1985. Using color to separate reflection components. Color Research & Application 10, 4 (1985), 210?218.
[36]
Jian Shi, Yue Dong, Hao Su, and Stella X Yu. 2017. Learning non-lambertian object intrinsics across shapenet categories. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1685?1694.
[37]
Zhixin Shu, Ersin Yumer, Sunil Hadap, Kalyan Sunkavalli, Eli Shechtman, and Dimitris Samaras. 2017. Neural face editing with intrinsic image disentangling. In Proceedings of the IEEE conference on computer vision and pattern recognition. 5541?5550.
[38]
Peter-Pike Sloan, Jan Kautz, and John Snyder. 2002. Precomputed radiance transfer for real-time rendering in dynamic, low-frequency lighting environments. In Proceedings of the 29th annual conference on Computer graphics and interactive techniques. 527?536.
[39]
Ayush Tewari, Michael Zollhofer, Hyeongwoo Kim, Pablo Garrido, Florian Bernard, Patrick Perez, and Christian Theobalt. 2017. Mofa: Model-based deep convolutional face autoencoder for unsupervised monocular reconstruction. In Proceedings of the IEEE International Conference on Computer Vision Workshops. 1274?1283.
[40]
Huiqiong Wang, S. Lin, Xiaopei Liu, and Sing Bing Kang. 2005. Separating reflections in human iris images for illumination estimation. In Tenth IEEE International Conference on Computer Vision (ICCV?05) Volume 1, Vol. 2. 1691?1698 Vol. 2. https://doi.org/10.1109/ICCV.2005.215
[41]
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. 2018. Non-local neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 7794?7803.
[42]
Felix Wimbauer, Shangzhe Wu, and Christian Rupprecht. 2022. De-rendering 3D Objects in the Wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 18490?18499.
[43]
Erroll Wood, Tadas Baltru?aitis, Charlie Hewitt, Sebastian Dziadzio, Thomas J Cashman, and Jamie Shotton. 2021. Fake it till you make it: face analysis in the wild using synthetic data alone. In Proceedings of the IEEE/CVF international conference on computer vision. 3681?3691.
[44]
Erroll Wood, Tadas Baltru?aitis, Louis-Philippe Morency, Peter Robinson, and Andreas Bulling. 2016a. A 3d morphable eye region model for gaze estimation. In European Conference on Computer Vision. Springer, 297?313.
[45]
Erroll Wood, Tadas Baltru?aitis, Louis-Philippe Morency, Peter Robinson, and Andreas Bulling. 2016b. Learning an appearance-based gaze estimator from one million synthesised images. In Proceedings of the Ninth Biennial ACM Symposium on Eye Tracking Research & Applications. 131?138.
[46]
Erroll Wood, Tadas Baltrusaitis, Xucong Zhang, Yusuke Sugano, Peter Robinson, and Andreas Bulling. 2015. Rendering of eyes for eye-shape registration and gaze estimation. In Proceedings of the IEEE international conference on computer vision. 3756?3764.
[47]
Shangzhe Wu, Christian Rupprecht, and Andrea Vedaldi. 2020. Unsupervised learning of probably symmetric deformable 3d objects from images in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1?10.
[48]
Jun Xu, Yingkun Hou, Dongwei Ren, Li Liu, Fan Zhu, Mengyang Yu, Haoqian Wang, and Ling Shao. 2020. Star: A structure and texture aware retinex model. IEEE Transactions on Image Processing 29 (2020), 5022?5037.
[49]
Yu-Ying Yeh, Koki Nagano, Sameh Khamis, Jan Kautz, Ming-Yu Liu, and Ting-Chun Wang. 2022. Learning to relight portrait images via a virtual light stage and synthetic-to-real adaptation. ACM Transactions on Graphics (TOG) 41, 6 (2022), 1?21.
[50]
Ye Yu and William AP Smith. 2019. Inverserendernet: Learning single image inverse rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 3155?3164.
[51]
Yongjie Zhu, Jiajun Tang, Si Li, and Boxin Shi. 2021. DeRenderNet: Intrinsic Image Decomposition of Urban Scenes with Shape-(In) dependent Shading Rendering. In 2021 IEEE International Conference on Computational Photography (ICCP). IEEE, 1?11.
