“Video to fully automatic 3D hair model”
Conference:
Type(s):
Title:
- Video to fully automatic 3D hair model
Session/Category Title: Modeling things on (and in) your head
Presenter(s)/Author(s):
Moderator(s):
Abstract:
Imagine taking a selfie video with your mobile phone and getting as output a 3D model of your head (face and 3D hair strands) that can be later used in VR, AR, and any other domain. State of the art hair reconstruction methods allow either a single photo (thus compromising 3D quality) or multiple views, but they require manual user interaction (manual hair segmentation and capture of fixed camera views that span full 360°). In this paper, we describe a system that can completely automatically create a reconstruction from any video (even a selfie video), and we don’t require specific views, since taking your -90°, 90°, and full back views is not feasible in a selfie capture.In the core of our system, in addition to the automatization components, hair strands are estimated and deformed in 3D (rather than 2D as in state of the art) thus enabling superior results. We provide qualitative, quantitative, and Mechanical Turk human studies that support the proposed system, and show results on a diverse variety of videos (8 different celebrity videos, 9 selfie mobile videos, spanning age, gender, hair length, type, and styling).
References:
1. Oleg Alexander, Graham Fyffe, Jay Busch, Xueming Yu, Ryosuke Ichikari, Andrew Jones, Paul Debevec, Jorge Jimenez, Etienne Danvoye, Bernardo Antionazzi, et al. 2013. Digital Ira: creating a real-time photoreal digital actor. In ACM SIGGRAPH 2013 Posters. ACM, 1. Google ScholarDigital Library
2. B. Allen, B. Curless, and Z. Popović. 2003. The space of human body shapes: reconstruction and parameterization from range scans. In ACM Transactions on Graphics (TOG), Vol. 22. ACM, 587–594. Google ScholarDigital Library
3. Thabo Beeler, Bernd Bickel, Paul Beardsley, Bob Sumner, and Markus Gross. 2010. High-quality single-shot capture of facial geometry. ACM Transactions on Graphics (TOG) 29, 4 (2010), 40. Google ScholarDigital Library
4. Volker Blanz and Thomas Vetter. 1999. A morphable model for the synthesis of 3D faces. In Proceedings of the 26th annual conference on Computer graphics and interactive techniques. ACM Press/Addison-Wesley Publishing Co., 187–194. Google ScholarDigital Library
5. Chen Cao, Yanlin Weng, Shun Zhou, Yiying Tong, and Kun Zhou. 2013. Facewarehouse: a 3d facial expression database for visual computing. (2013).Google Scholar
6. Chen Cao, Hongzhi Wu, Yanlin Weng, Tianjia Shao, and Kun Zhou. 2016. Real-time facial animation with image-based dynamic avatars. ACM Transactions on Graphics (TOG) 35, 4 (2016), 126. Google ScholarDigital Library
7. Menglei Chai, Linjie Luo, Kalyan Sunkavalli, Nathan Carr, Sunil Hadap, and Kun Zhou. 2015. High-quality hair modeling from a single portrait photo. ACM Transactions on Graphics (TOG) 34, 6 (2015), 204. Google ScholarDigital Library
8. Menglei Chai, Tianjia Shao, Hongzhi Wu, Yanlin Weng, and Kun Zhou. 2016. Autohair: Fully automatic hair modeling from a single image. ACM Transactions on Graphics (TOG) 35, 4 (2016), 116. Google ScholarDigital Library
9. Menglei Chai, Lvdi Wang, Yanlin Weng, Xiaogang Jin, and Kun Zhou. 2013. Dynamic hair manipulation in images and videos. ACM Transactions on Graphics (TOG) 32, 4 (2013), 75. Google ScholarDigital Library
10. Menglei Chai, Lvdi Wang, Yanlin Weng, Yizhou Yu, Baining Guo, and Kun Zhou. 2012. Single-view hair modeling for portrait manipulation. ACM Transactions on Graphics (TOG) 31, 4 (2012), 116. Google ScholarDigital Library
11. Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. 2016. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. arXiv preprint arXiv:1606.00915 (2016).Google Scholar
12. Paul Debevec. 2012. The light stages and their applications to photoreal digital actors. SIGGRAPH Asia (2012).Google Scholar
13. Hongbo Fu, Yichen Wei, Chiew-Lan Tai, and Long Quan. 2007. Sketching hairstyles. In Proceedings of the 4th Eurographics workshop on Sketch-based interfaces and modeling. ACM, 31–36. Google ScholarDigital Library
14. Michael Goesele, Noah Snavely, Brian Curless, Hugues Hoppe, and Steven M Seitz. 2007. Multi-view stereo for community photo collections. In Computer Vision, 2007. ICCV 2007. IEEE 11th International Conference on. IEEE, 1–8.Google ScholarCross Ref
15. Hairbobo. 2017. Hairbobo. http://www.hairbobo.com/faxingtupian. (Sept. 2017).Google Scholar
16. Liwen Hu, Chongyang Ma, Linjie Luo, and Hao Li. 2014a. Robust hair capture using simulated examples. ACM Transactions on Graphics (TOG) 33, 4 (2014), 126. Google ScholarDigital Library
17. Liwen Hu, Chongyang Ma, Linjie Luo, and Hao Li. 2015. Single-view hair modeling using a hairstyle database. ACM Transactions on Graphics (TOG) 34, 4 (2015), 125. Google ScholarDigital Library
18. Liwen Hu, Chongyang Ma, Linjie Luo, Li-Yi Wei, and Hao Li. 2014b. Capturing braided hairstyles. ACM Transactions on Graphics (TOG) 33, 6 (2014), 225. Google ScholarDigital Library
19. Liwen Hu, Shunsuke Saito, Lingyu Wei, Koki Nagano, Jaewoo Seo, Jens Fursund, Iman Sadeghi, Carrie Sun, Yen-Chun Chen, and Hao Li. 2017. Avatar Digitization from a Single Image for Real-time Rendering. ACM Trans. Graph. 36, 6, Article 195 (Nov. 2017), 14 pages. Google ScholarDigital Library
20. Alexandru Eugen Ichim, Sofien Bouaziz, and Mark Pauly. 2015. Dynamic 3D avatar creation from hand-held video input. ACM Transactions on Graphics (TOG) 34, 4 (2015), 45. Google ScholarDigital Library
21. Wenzel Jakob, Jonathan T Moon, and Steve Marschner. 2009. Capturing hair assemblies fiber by fiber. In ACM Transactions on Graphics (TOG), Vol. 28. ACM, 164. Google ScholarDigital Library
22. Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. 2014. Caffe: Convolutional architecture for fast feature embedding. In Proceedings of the 22nd ACM international conference on Multimedia. ACM, 675–678. Google ScholarDigital Library
23. Michael Kazhdan and Hugues Hoppe. 2013. Screened poisson surface reconstruction. ACM Transactions on Graphics (TOG) 32, 3 (2013), 29. Google ScholarDigital Library
24. Ira Kemelmacher-Shlizerman. 2016. Transfiguring portraits. ACM Transactions on Graphics (TOG) 35, 4 (2016), 94. Google ScholarDigital Library
25. Ira Kemelmacher-Shlizerman and Ronen Basri. 2011. 3d face reconstruction from a single image using a single reference face shape. Pattern Analysis and Machine Intelligence, IEEE Transactions on 33, 2 (2011), 394–405. Google ScholarDigital Library
26. Ira Kemelmacher-Shlizerman and Steven M Seitz. 2011. Face reconstruction in the wild. In Computer Vision (ICCV), 2011 IEEE International Conference on. IEEE, 1746–1753. Google ScholarDigital Library
27. Aldo Laurentini. 1994. The visual hull concept for silhouette-based image understanding. IEEE Transactions on pattern analysis and machine intelligence 16, 2 (1994), 150–162. Google ScholarDigital Library
28. Shu Liang, Ira Kemelmacher-Shlizerman, and Linda G Shapiro. 2014. 3d face hallucination from a single depth frame. In 3D Vision (3DV), 2014 2nd international conference on, Vol. 1. IEEE, 31–38. Google ScholarDigital Library
29. Shu Liang, Linda G Shapiro, and Ira Kemelmacher-Shlizerman. 2016. Head reconstruction from internet photos. In European Conference on Computer Vision. Springer, 360–374.Google ScholarCross Ref
30. Sifei Liu, Jianping Shi, Ji Liang, and Ming-Hsuan Yang. 2017. Face Parsing via Recurrent Propagation. arXiv preprint arXiv:1708.01936 (2017).Google Scholar
31. Sifei Liu, Jimei Yang, Chang Huang, and Ming-Hsuan Yang. 2015. Multi-objective convolutional learning for face labeling. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3451–3459.Google Scholar
32. Jonathan Long, Evan Shelhamer, and Trevor Darrell. 2015. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3431–3440.Google ScholarCross Ref
33. Linjie Luo, Hao Li, and Szymon Rusinkiewicz. 2013. Structure-aware hair capture. ACM Transactions on Graphics (TOG) 32, 4 (2013), 76. Google ScholarDigital Library
34. Ping Luo, Xiaogang Wang, and Xiaoou Tang. 2012. Hierarchical face parsing via deep learning. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on. IEEE, 2480–2487. Google ScholarDigital Library
35. Fabio Maninchedda, Christian Häne, Bastien Jacquet, Amaël Delaunoy, and Marc Pollefeys. 2016. Semantic 3D Reconstruction of Heads. In European Conference on Computer Vision. Springer, 667–683.Google ScholarCross Ref
36. Richard A Newcombe, Dieter Fox, and Steven M Seitz. 2015. DynamicFusion: Reconstruction and tracking of non-rigid scenes in real-time. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 343–352.Google ScholarCross Ref
37. Sylvain Paris, Hector M Briceño, and François X Sillion. 2004. Capture of hair geometry from multiple images. In ACM Transactions on Graphics (TOG), Vol. 23. ACM, 712–719. Google ScholarDigital Library
38. Sylvain Paris, Will Chang, Oleg I Kozhushnyan, Wojciech Jarosz, Wojciech Matusik, Matthias Zwicker, and Frédo Durand. 2008. Hair photobooth: geometric and photometric acquisition of real hairstyles. In ACM Transactions on Graphics (TOG), Vol. 27. ACM, 30. Google ScholarDigital Library
39. Elad Richardson, Matan Sela, and Ron Kimmel. 2016. 3D face reconstruction by learning from synthetic data. In 3D Vision (3DV), 2016 Fourth International Conference on. IEEE, 460–469.Google Scholar
40. Elad Richardson, Matan Sela, Roy Or-El, and Ron Kimmel. 2017. Learning detailed face reconstruction from a single image. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 5553–5562.Google ScholarCross Ref
41. Shunsuke Saito, Lingyu Wei, Liwen Hu, Koki Nagano, and Hao Li. 2016. Photorealistic Facial Texture Inference Using Deep Neural Networks. arXiv preprint arXiv:1612.00523 (2016).Google Scholar
42. Olga Sorkine, Daniel Cohen-Or, Yaron Lipman, Marc Alexa, Christian Rössl, and H-P Seidel. 2004. Laplacian surface editing. In Proceedings of the 2004 Eurographics/ACM SIGGRAPH symposium on Geometry processing. ACM, 175–184. Google ScholarDigital Library
43. Supasorn Suwajanakorn, Ira Kemelmacher-Shlizerman, and Steven M Seitz. 2014. Total moving face reconstruction. In Computer Vision-ECCV 2014.Google ScholarCross Ref
44. Supasorn Suwajanakorn, Steven M Seitz, and Ira Kemelmacher-Shlizerman. 2015. What Makes Tom Hanks Look Like Tom Hanks. In Proceedings of the IEEE International Conference on Computer Vision. 3952–3960. Google ScholarDigital Library
45. Justus Thies, Michael Zollhoefer, Matthias Niessner, Levi Valgaerts, Marc Stamminger, and Christian Theobalt. 2015. Real-time Expression Transfer for Facial Reenactment. ACM Transactions on Graphics (Proc. SIGGRAPH Asia) (2015). Google ScholarDigital Library
46. Anh Tuan Tran, Tal Hassner, Iacopo Masi, and Gérard Medioni. 2017. Regressing robust and discriminative 3D morphable models with a very deep neural network. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 1493–1502.Google ScholarCross Ref
47. Nuttapon Vanakittistien, Attawith Sudsang, and Nuttapong Chentanez. 2016. 3D hair model from small set of images. In Proceedings of the 9th International Conference on Motion in Games. ACM, 85–90. Google ScholarDigital Library
48. Dan Wang, Xiujuan Chai, Hongming Zhang, Hong Chang, Wei Zeng, and Shiguang Shan. 2011. A novel coarse-to-fine hair segmentation method. In Automatic Face & Gesture Recognition and Workshops (FG 2011), 2011 IEEE International Conference on. IEEE, 233–238.Google ScholarCross Ref
49. Kelly Ward, Florence Bertails, Tae-Yong Kim, Stephen R Marschner, Marie-Paule Cani, and Ming C Lin. 2007. A survey on hair modeling: Styling, simulation, and rendering. IEEE Transactions on Visualization and Computer Graphics 13, 2 (2007). Google ScholarDigital Library
50. Yanlin Weng, Lvdi Wang, Xiao Li, Menglei Chai, and Kun Zhou. 2013. Hair interpolation for portrait morphing. In Computer Graphics Forum, Vol. 32. Wiley Online Library, 79–84.Google Scholar
51. Changchang Wu. 2011. VisualSFM: A visual structure from motion system. (2011).Google Scholar
52. Xuehan Xiong and Fernando De la Torre. 2013. Supervised Descent Method and its Applications to Face Alignment. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Google ScholarDigital Library
53. Yaser Yacoob and Larry S Davis. 2006. Detection and analysis of hair. IEEE transactions on pattern analysis and machine intelligence 28, 7 (2006), 1164–1169. Google ScholarDigital Library
54. Meng Zhang, Menglei Chai, Hongzhi Wu, Hao Yang, and Kun Zhou. 2017. Adata-driven approach to four-view image-based hair modeling. ACM Transactions on Graphics (TOG) 36, 4 (2017), 156. Google ScholarDigital Library
55. Shuai Zheng, Sadeep Jayasumana, Bernardino Romera-Paredes, Vibhav Vineet, Zhizhong Su, Dalong Du, Chang Huang, and Philip HS Torr. 2015. Conditional random fields as recurrent neural networks. In Proceedings of the IEEE International Conference on Computer Vision. 1529–1537. Google ScholarDigital Library
56. Michael Zollhöfer, Matthias Nießner, Shahram Izadi, Christoph Rehmann, Christopher Zach, Matthew Fisher, Chenglei Wu, Andrew Fitzgibbon, Charles Loop, Christian Theobalt, et al. 2014. Real-time non-rigid reconstruction using an rgb-d camera. ACM Transactions on Graphics, TOG (2014). Google ScholarDigital Library
