“Physics-Inspired Garment Recovery from a Single-View Image” by Yang, Pan, Amert, Wang, Yu, et al. …

  • ©Shan Yang, Zherong Pan, Tanya Amert, Ke Wang, Licheng Yu, Tamara Berg, and Ming C. Lin

Conference:


Type:


Title:

    Physics-Inspired Garment Recovery from a Single-View Image

Session/Category Title: Textiles & Microstructures

Presenter(s)/Author(s):


Moderator(s):


Abstract:


    Most recent garment capturing techniques rely on acquiring multiple views of clothing, which may not always be readily available, especially in the case of pre-existing photographs from the web. As an alternative, we propose a method that is able to compute a 3D model of a human body and its outfit from a single photograph with little human interaction. Our algorithm is not only able to capture the global shape and overall geometry of the clothing, it can also extract the physical properties (i.e., material parameters needed for simulation) of cloth. Unlike previous methods using full 3D information (i.e., depth, multi-view images, or sampled 3D geometry), our approach achieves garment recovery from a single-view image by using physical, statistical, and geometric priors and a combination of parameter estimation, semantic parsing, shape/pose recovery, and physics-based cloth simulation. We demonstrate the effectiveness of our algorithm by re-purposing the reconstructed garments for virtual try-on and garment transfer applications and for cloth animation on digital characters.

References:


    1. Ankur Agarwal and Bill Triggs. 2006. Recovering 3D human pose from monocular images. IEEE Transactions on Pattern Analysis and Machine Intelligence 28, 1 (2006), 44–58. 
    2. AliExpress. 2015. Homepage. Retrieved from http://www.aliexpress.com.
    3. Dragomir Anguelov, Praveen Srinivasan, Daphne Koller, Sebastian Thrun, Jim Rodgers, and James Davis. 2005. SCAPE: Shape completion and animation of people. ACM Transactions on Graphics (TOG) 24, 3 (2005), 408–416. 
    4. Anthropologie. 2015. Homepage. Retrieved from http://www.anthropologie.com.
    5. Alexandru O. Balan, Leonid Sigal, Michael J. Black, James E. Davis, and Horst W. Haussecker. 2007. Detailed human shape and pose from images. In Proceedings of the 2007 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 1–8.
    6. David Baraff and Andrew Witkin. 1998. Large steps in cloth simulation. In Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques. ACM, 43–54. 
    7. Jo Barnfield. 2012. The Pattern Making Primer: All You Need to Know About Designing, Adapting and Customizing Sewing Patterns. Barron’s Educational Series, Inc, Hauppauge, NY.
    8. Aric Bartle, Alla Sheffer, Vladimir G. Kim, Danny M. Kaufman, Nicholas Vining, and Floraine Berthouzoz. 2016. Physics-driven pattern adjustment for direct 3D garment editing. ACM Transactions on Graphics (TOG) 35, 4 (2016), 50–1. 
    9. Pascal Bérard, Derek Bradley, Maurizio Nitti, Thabo Beeler, and Markus Gross. 2014. High-quality capture of eyes. ACM Transactions on Graphics (TOG) 33, 6 (2014), 223. 
    10. Floraine Berthouzoz, Akash Garg, Danny M. Kaufman, Eitan Grinspun, and Maneesh Agrawala. 2013. Parsing sewing patterns into 3D garments. ACM Transactions on Graphics (TOG) 32, 4 (2013), 85. 
    11. Boden. 2015. Homepage. Retrieved from http://www.bodenusa.com.
    12. Robert Bridson, Ronald Fedkiw, and John Anderson. 2002. Robust treatment of collisions, contact and friction for cloth animation. ACM Transactions on Graphics (TOG) 21, 3 (2002), 594–603. 
    13. Robert Bridson, Sebastian Marino, and Ronald Fedkiw. 2003. Simulation of clothing with folds and wrinkles. In Proceedings of the 2003 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. Eurographics Association, 28–36. 
    14. Alexander M. Bronstein, Michael M Bronstein, and Ron Kimmel. 2006. Efficient computation of isometry-invariant distances between surfaces. SIAM Journal on Scientific Computing 28, 5 (2006), 1812–1836.
    15. Alexander M. Bronstein, Michael M. Bronstein, and Ron Kimmel. 2008. Numerical Geometry of Non-rigid Shapes. Springer Science 8 Business Media. 
    16. Remi Brouet, Alla Sheffer, Laurence Boissieux, and Marie-Paule Cani. 2012. Design preserving garment transfer. ACM Transactions on Graphics 31, 4 (2012), 36. 
    17. Chen Cao, Yanlin Weng, Stephen Lin, and Kun Zhou. 2013. 3D shape regression for real-time facial animation. ACM Transactions on Graphics (TOG) 32, 4 (2013), 41. 
    18. Romain Casati, Gilles Daviet, and Florence Bertails-Descoubes. 2016. Inverse Elastic Cloth Design with Contact and Friction. Ph.D. Dissertation. Inria Grenoble Rhône-Alpes, Université de Grenoble.
    19. Menglei Chai, Lvdi Wang, Yanlin Weng, Yizhou Yu, Baining Guo, and Kun Zhou. 2012. Single-view hair modeling for portrait manipulation. ACM Transactions on Graphics 31, 4 (July 2012), Article 116, 8 pages. 
    20. Xiaowu Chen, Yu Guo, Bin Zhou, and Qinping Zhao. 2013. Deformable model for estimating clothed and naked human shapes from a single image. The Visual Computer 29, 11 (2013), 1187–1196. 
    21. Xiaowu Chen, Bin Zhou, Feixiang Lu, Lin Wang, Lang Bi, and Ping Tan. 2015. Garment modeling with a depth camera. ACM Transactions on Graphics (TOG) 34, 6 (2015), 203. 
    22. Sean Curtis, Rasmus Tamstorf, and Dinesh Manocha. 2008. Fast collision detection for deformable models using representative-triangles. In Proceedings of the 2008 Symposium on Interactive 3D Graphics and Games. ACM, 61–69. 
    23. Philippe Decaudin, Dan Julius, Jamie Wither, Laurence Boissieux, Alla Sheffer, and Marie-Paule Cani. 2006. Virtual garments: A fully geometric approach for clothing design. Computer Graphics Forum 25, 625–634.
    24. Elliot English and Robert Bridson. 2008. Animating developable surfaces using nonconforming elements. ACM Transactions on Graphics (TOG) 27, 3 (2008), 66. 
    25. C. Farabet, C. Couprie, L. Najman, and Y. LeCun. 2013. Learning hierarchical features for scene labeling. Pattern Analysis and Machine Intelligence 35, 8 (2013), 1915–1929. 
    26. FashionableShoes. 2013. Homepage. Retrieved from http://bestfashionableshoess.blogspot.com.
    27. FashionUnited. 2016. Global Fashion Industry Statistics—International Apparel. https://fashionunited.com/global-fashion-industry-statistics.
    28. Rony Goldenthal, David Harmon, Raanan Fattal, Michel Bercovier, and Eitan Grinspun. 2007. Efficient simulation of inextensible cloth. ACM Transactions on Graphics (TOG) 26, 3 (2007), 49. 
    29. Naga K. Govindaraju, Ilknur Kabul, Ming C. Lin, and Dinesh Manocha. 2007. Fast continuous collision detection among deformable models using graphics processors. Computers 8 Graphics 31, 1 (2007), 5–14. 
    30. Nils Hasler, Mark Asbach, Bodo Rosenhahn, Jens-Rainer Ohm, and Hans-Peter Seidel. 2006. Physically based tracking of cloth. In Proceedings of the International Workshop on Vision, Modeling, and Visualization, VMV. 49–56.
    31. Nils Hasler, Carsten Stoll, Martin Sunkel, Bodo Rosenhahn, and H.-P. Seidel. 2009. A statistical model of human pose and body shape. Computer Graphics Forum 28, 337–346.
    32. Hillsweddingdress. 2015. Homepage. Retrieved from http://hillsweddingdress.xyz.
    33. Donald H. House and David E. Breen. 2000. Cloth Modeling and Animation. AK Peters. 
    34. Takeo Igarashi, Tomer Moscovich, and John F. Hughes. 2005. As-rigid-as-possible shape manipulation. ACM Transactions on Graphics (TOG) 24, 3 (2005), 1134–1141. 
    35. Michal Jancosek and Tomás Pajdla. 2011. Multi-view reconstruction preserving weakly-supported surfaces. In Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 3121–3128. 
    36. Moon-Hwan Jeong, Dong-Hoon Han, and Hyeong-Seok Ko. 2015. Garment capture from a photograph. Computer Animation and Virtual Worlds 26, 3–4 (2015), 291–300. 
    37. Ladislav Kavan, P.-P. Sloan, and Carol O’Sullivan. 2010. Fast and efficient skinning of animated meshes. Computer Graphics Forum 29, 327–336.
    38. James Kennedy. 2010. Particle swarm optimization. In Encyclopedia of Machine Learning. Springer, 760–766.
    39. Anat Levin, Dani Lischinski, and Yair Weiss. 2008. A closed-form solution to natural image matting. IEEE Transactions on Pattern Analysis and Machine Intelligence 30, 2 (2008), 228–242. 
    40. Hao Li, Linjie Luo, Daniel Vlasic, Pieter Peers, Jovan Popović, Mark Pauly, and Szymon Rusinkiewicz. 2012. Temporally coherent completion of dynamic shapes. ACM Transactions on Graphics 31, 1 (January 2012), 2. 
    41. Yin Li, Jian Sun, Chi-Keung Tang, and Heung-Yeung Shum. 2004. Lazy snapping. In ACM Transactions on Graphics (TOG) 23, 3 (2004), 303–308. 
    42. Matt Lindner. 2015. Global E-commerce Sales Set to Grow 25% in 2015. Retrieved from https://www.internetretailer.com/2015/07/29/global-e-commerce-set-grow-25-2015.
    43. Jonathan Long, Evan Shelhamer, and Trevor Darrell. 2015. Fully convolutional networks for semantic segmentation. CVPR (to Appear) (Nov. 2015). arxiv:cs.CV/1411.4038
    44. Yuwei Meng, Charlie C. L. Wang, and Xiaogang Jin. 2012. Flexible shape control for automatic resizing of apparel products. Computer-Aided Design 44, 1 (2012), 68–76. 
    45. ModCloth. 2015. Homepage. Retrieved from http://www.modcloth.com.
    46. Thomas B. Moeslund, Adrian Hilton, and Volker Krüger. 2006. A survey of advances in vision-based human motion capture and analysis. Computer Vision and Image Understanding 104, 2 (2006), 90–126. 
    47. Koki Nagano, Graham Fyffe, Oleg Alexander, Jernej Barbiç, Hao Li, Abhijeet Ghosh, and Paul Debevec. 2015. Skin microstructure deformation with displacement map convolution. ACM Transactions on Graphics 34, 4, Article 109 (July 2015), 10 pages. 
    48. Rahul Narain, Armin Samii, and James F. O’Brien. 2012. Adaptive anisotropic remeshing for cloth simulation. ACM Transactions on Graphics (TOG) 31, 6 (2012), 152. 
    49. Hing N. Ng and Richard L. Grimsdale. 1996. Computer graphics techniques for modeling cloth. IEEE Computer Graphics and Applications 16, 5 (1996), 28–41. 
    50. P. H. Pinheiro and R. Collobert. 2014. Recurrent convolutional neural networks for scene labeling. In ICML. 
    51. Tiberiu Popa, Quan Zhou, Derek Bradley, Vladislav Kraevoy, Hongbo Fu, Alla Sheffer, and Wolfgang Heidrich. 2009. Wrinkling captured garments using space-time data-driven deformation. Computer Graphics Forum 28, 427–435.
    52. Dimitris Protopsaltou, Christiane Luible, Marlene Arevalo, and Nadia Magnenat-Thalmann. 2002. A Body and Garment Creation Method for an Internet Based Virtual Fitting Room. Springer.
    53. RedBubble. 2015. Homepage. Retrieved from http://www.redbubble.com.
    54. Cody Robson, Ron Maharik, Alla Sheffer, and Nathan Carr. 2011. Context-aware garment modeling from sketches. Computers 8 Graphics 35, 3 (2011), 604–613. 
    55. Damien Rohmer, Tiberiu Popa, Marie-Paule Cani, Stefanie Hahmann, and Alla Sheffer. 2010. Animation wrinkling: Augmenting coarse cloth simulations with realistic-looking wrinkles. In ACM Transactions on Graphics (TOG) 29, 6 (2010), 157. 
    56. Saaclothes. 2015. Retrieved from http://www.saaclothes.com.
    57. Volker Scholz and Marcus Magnor. 2006. Texture replacement of garments in monocular video sequences. In Proceedings of the 17th Eurographics Conference on Rendering Techniques. Eurographics Association, 305–312. 
    58. Volker Scholz, Timo Stich, Michael Keckeisen, Markus Wacker, and Marcus Magnor. 2005. Garment motion capture using color-coded patterns. Computer Graphics Forum 24, 439–447.
    59. Hyewon Seo and Nadia Magnenat-Thalmann. 2003. An automatic modeling of human bodies from sizing parameters. In Proceedings of the 2003 Symposium on Interactive 3D Graphics. ACM, 19–26. 
    60. Leonid Sigal, Moshe Mahler, Spencer Diaz, Kyna McIntosh, Elizabeth Carter, Timothy Richards, and Jessica Hodgins. 2015. A perceptual control space for garment simulation. ACM Transactions on Graphics (TOG) 34, 4 (2015), 117. 
    61. Robert W. Sumner and Jovan Popović. 2004. Deformation transfer for triangle meshes. ACM Transactions on Graphics (TOG) 23, 3 (2004), 399–405. 
    62. Min Tang, Sean Curtis, Sung-Eui Yoon, and Dinesh Manocha. 2009. ICCD: Interactive continuous collision detection between deformable models using connectivity-based culling. IEEE Transactions on Visualization and Computer Graphics 15, 4 (2009), 544–557. 
    63. Hiroaki Tanie, Katsu Yamane, and Yoshihiko Nakamura. 2005. High marker density motion capture by retroreflective mesh suit. In Proceedings of the 2005 IEEE International Conference on Robotics and Automation. ICRA 2005. IEEE, 2884–2889.
    64. Gabriel Taubin. 1995. A signal processing approach to fair surface design. In Proceedings of the 22nd Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH’95). ACM, New York, NY, 351–358. 
    65. Camillo J. Taylor. 2000. Reconstruction of articulated objects from point correspondences in a single uncalibrated image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Vol. 1. IEEE, 677–684.
    66. Bernhard Thomaszewski, Simon Pabst, and Wolfgang Strasser. 2009. Continuum-based strain limiting. Computer Graphics Forum 28, 569–576.
    67. Emmanuel Turquin, Jamie Wither, Laurence Boissieux, Marie-Paule Cani, and John F Hughes. 2007. A sketch-based interface for clothing virtual characters. IEEE Computer Graphics and Applications1 (2007), 72–81. 
    68. Frans Van den Bergh and Andries Petrus Engelbrecht. 2006. A study of particle swarm optimization particle trajectories. Information sciences 176, 8 (2006), 937–971. 
    69. Pascal Volino and Nadia Magnenat-Thalmann. 1999. Fast geometrical wrinkles on animated surfaces. In Proceedings of the 7th International Conference in Central Europe on Computer Graphics and Visualization (Winter School on Computer Graphics).
    70. Charlie CL Wang, Yu Wang, and Matthew MF Yuen. 2005. Design automation for customized apparel products. Computer-Aided Design 37, 7 (2005), 675–691. 
    71. Huamin Wang, James O’Brien, and Ravi Ramamoorthi. 2010. Multi-resolution isotropic strain limiting. ACM Transactions on Graphics (TOG) 29, 156. 
    72. Huamin Wang, James F. O’Brien, and Ravi Ramamoorthi. 2011. Data-driven elastic models for cloth: Modeling and measurement. ACM Transactions on Graphics (TOG) 30, 4 (2011), 71. 
    73. Jerry Weil. 1986. The synthesis of cloth objects. ACM Siggraph Computer Graphics 20, 4 (1986), 49–54. 
    74. Ryan White, Keenan Crane, and David A. Forsyth. 2007. Capturing and animating occluded cloth. ACM Transactions on Graphics (TOG), Vol. 26. ACM, 34. 
    75. Changchang Wu. 2011. VisualSFM: A visual structure from motion system. Retreived from http://homes.cs.washington.edu/ccwu/vsfm 9 (2011).
    76. Changchang Wu. 2013. Towards linear-time incremental structure from motion. In Proceedings of the 2013 International Conference on 3D Vision. 127–134. 
    77. Saining Xie and Zhuowen Tu. 2015. Holistically-nested edge detection. In Proceedings of the IEEE International Conference on Computer Vision. 1395–1403. 
    78. Kazuhiro Yamaguchi, Mohammad Hadi Kiapour, and Tamara Berg. 2013. Paper doll parsing: Retrieving similar styles to parse clothing items. In Proceedings of the 2013 IEEE International Conference on Computer Vision. IEEE, 3519–3526. 
    79. Yipin Yang, Yao Yu, Yu Zhou, Sidan Du, James Davis, and Ruigang Yang. 2014. Semantic parametric reshaping of human body models. In Proceedings of the 2014 2nd International Conference on 3D Vision, Vol. 2. IEEE, 41–48. 
    80. Mao Ye, Huamin Wang, Nianchen Deng, Xubo Yang, and Ruigang Yang. 2014. Real-time human pose and shape estimation for virtual try-on using a single commodity depth camera. IEEE Transactions on Visualization and Computer Graphics 20, 4 (2014), 550–559. 
    81. S. D. Young, B. D. Adelstein, and S. R. Ellis. 2007. Calculus of nonrigid surfaces for geometry and texture manipulation. IEEE Transactions on Visualization and Computer Graphics 13, 5 (2007), 902–913. 
    82. Wenyi Zhao, Rama Chellappa, P. Jonathon Phillips, and Azriel Rosenfeld. 2003. Face recognition: A literature survey. ACM Computing Surveys (CSUR) 35, 4 (2003), 399–458. 
    83. Bin Zhou, Xiaowu Chen, Qiang Fu, Kan Guo, and Ping Tan. 2013. Garment modeling from a single image. Computer Graphics Forum 32, 85–91.
    84. Shizhe Zhou, Hongbo Fu, Ligang Liu, Daniel Cohen-Or, and Xiaoguang Han. 2010. Parametric reshaping of human bodies in images. ACM Transactions on Graphics (TOG), Vol. 29. ACM, 126. 

ACM Digital Library Publication:


Overview Page: