“NeuralTailor: reconstructing sewing pattern structures from 3D point clouds of garments” by Korosteleva and Lee

The fields of SocialVR, performance capture, and virtual try-on are often faced with a need to faithfully reproduce real garments in the virtual world. One critical task is the disentanglement of the intrinsic garment shape from deformations due to fabric properties, physical forces, and contact with the body. We propose to use a garment sewing pattern, a realistic and compact garment descriptor, to facilitate the intrinsic garment shape estimation. Another major challenge is a high diversity of shapes and designs in the domain. The most common approach for Deep Learning on 3D garments is to build specialized models for individual garments or garment types. We argue that building a unified model for various garment designs has the benefit of generalization to novel garment types, hence covering a larger design domain than individual models would. We introduce NeuralTailor, a novel architecture based on point-level attention for set regression with variable cardinality, and apply it to the task of reconstructing 2D garment sewing patterns from the 3D point cloud garment models. Our experiments show that NeuralTailor successfully reconstructs sewing patterns and generalizes to garment types with pattern topologies unseen during training.

1. Seungbae Bang, Maria Korosteleva, and Sung-Hee Lee. 2021. Estimating Garment Patterns from Static Scan Data. Computer Graphics Forum 40, 6 (2021), 273–287. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/cgf.14272 Google ScholarCross Ref
2. Aric Bartle, Alla Sheffer, Vladimir G. Kim, Danny M. Kaufman, Nicholas Vining, and Floraine Berthouzoz. 2016. Physics-driven pattern adjustment for direct 3D garment editing. ACM Transactions on Graphics 35, 4 (2016), 50–1. Google ScholarDigital Library
3. Hugo Bertiche, Meysam Madadi, and Sergio Escalera. 2020. CLOTH3D: Clothed 3D Humans. In European Conference on Computer Vision, Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (Eds.). Springer International Publishing, Virtual, 344–359. Google ScholarCross Ref
4. Lukas Biewald. 2020. Experiment Tracking with Weights and Biases. https://www.wandb.com/ Software available from wandb.com.Google Scholar
5. Remi Brouet, Alla Sheffer, Laurence Boissieux, and Marie-Paule Cani. 2012. Design preserving garment transfer. ACM Transactions on Graphics 31, 4 (7 2012), 1–11. Google ScholarDigital Library
6. Alexandre Carlier, Martin Danelljan, Alexandre Alahi, and Radu Timofte. 2020. DeepSVG: A hierarchical generative network for vector graphics animation. In Advances in Neural Information Processing Systems, Vol. 33. Curran Associates, Inc., 16351–16361.Google Scholar
7. Siddhartha Chaudhuri, Iit Bombay ERSIN YUMER, Adobe Research HAO ZHANG, Jun Li, Kai Xu, Ersin Yumer, and Hao Zhang. 2017. GRASS: Generative Recursive Autoencoders for Shape Struc-tures. ACM Transactions on Graphics (TOG) 36, 4 (2017), 1–14. Google ScholarDigital Library
8. Xiaowu Chen, Bin Zhou, Feixiang Lu, Lin Wang, Lang Bi, and Ping Tan. 2015. Garment modeling with a depth camera. ACM Transactions on Graphics 34, 6 (2015), 1–12. Google ScholarDigital Library
9. Enric Corona, Albert Pumarola, Guillem Alenya, Gerard Pons-Moll, and Francesc Moreno-Noguer. 2021. SMPLicit: Topology-Aware Generative Model for Clothed People. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Virtual, 11875–11885.Google ScholarCross Ref
10. Hein Daanen and Sung Ae Hong. 2008. Made-to-measure pattern development based on 3D whole body scans. International Journal of Clothing Science and Technology 20, 1 (2008), 15–25. Google ScholarCross Ref
11. Philippe Decaudin, Dan Julius, Jamie Wither, Laurence Boissieux, Alla Sheffer, and Marie Paule Cani. 2006. Virtual garments: A fully geometric approach for clothing design. Computer Graphics Forum 25, 3 (2006), 625–634. Google ScholarCross Ref
12. Matthias Fey and Jan E. Lenssen. 2019. Fast Graph Representation Learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds.Google Scholar
13. Chihiro Goto and Nobuyuki Umetani. 2021. Data-driven Garment Pattern Estimation from 3D Geometries. In Eurographics 2021 – Short Papers. The Eurographics Association, 17–20. Google ScholarCross Ref
14. Yulan Guo, Hanyun Wang, Qingyong Hu, Hao Liu, Li Liu, and Mohammed Bennamoun. 2020. Deep Learning for 3D Point Clouds: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence (12 2020), 1–1. Google ScholarDigital Library
15. David Ha and Douglas Eck. 2018. A neural representation of sketch drawings. In 6th International Conference on Learning Representations, ICLR 2018 – Conference Track Proceedings.Google Scholar
16. Nils Hasler, Bodo Rosenhahn, and Hans Peter Seidel. 2007. Reverse engineering garments. In International Conference on Computer Vision/Computer Graphics Collaboration Techniques and Applications. Springer, Berlin, Heidelberg, 200–211. Google ScholarCross Ref
17. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Computation 9, 8 (11 1997), 1735–1780. Google ScholarDigital Library
18. Takeo Igarashi, Tomer Moscovich, and John F. Hughes. 2005. As-rigid-as-possible shape manipulation. ACM Transactions on Graphics (TOG) 24, 3 (7 2005), 1134–1141. Google ScholarDigital Library
19. Moon-Hwan Jeong, Dong-Hoon Han, and Hyeong-Seok Ko. 2015. Garment capture from a photograph. Computer Animation and Virtual Worlds 26, 3–4 (2015), 291–300. Google ScholarDigital Library
20. Boyi Jiang, Juyong Zhang, Yang Hong, Jinhao Luo, Ligang Liu, and Hujun Bao. 2020. BCNet: Learning Body and Cloth Shape from a Single Image. In Computer Vision – ECCV 2020. Springer International Publishing, Cham, 18–35. Google ScholarCross Ref
21. Diederik P. Kingma and Jimmy Lei Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations (ICLR). International Conference on Learning Representations, ICLR, San-Diego, USA, abs/1412.6980. https://arxiv.org/abs/1412.6980v9Google Scholar
22. Maria Korosteleva and Sung-Hee Lee. 2021a. Dataset of 3D Garments with Sewing Patterns. Google ScholarCross Ref
23. Maria Korosteleva and Sung-Hee Lee. 2021b. Generating Datasets of 3D Garments with Sewing Patterns. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, J. Vanschoren and S. Yeung (Eds.), Vol. 1. Virtual.Google Scholar
24. Bharat Lal Bhatnagar, Garvita Tiwari, Christian Theobalt, and Gerard Pons-Moll. 2019. Multi-Garment Net: Learning to Dress 3D People from Images. In The IEEE International Conference on Computer Vision (ICCV). IEEE, Seoul, Republic of Korea, 5420–5430.Google ScholarCross Ref
25. Minchen Li, Alla Sheffer, Eitan Grinspun, and Nicholas Vining. 2018. FoldSketch: Enriching garments with physically reproducible folds. ACM Transactions on Graphics 37, 4 (8 2018), 1–13. Google ScholarDigital Library
26. Kaixuan Liu, Xianyi Zeng, Pascal Bruniaux, Xuyuan Tao, Xiaofeng Yao, Victoria Li, and Jianping Wang. 2018. 3D interactive garment pattern-making technology. CAD Computer Aided Design 104 (11 2018), 113–124. Google ScholarDigital Library
27. Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-moll, and Michael J Black. 2015. SMPL: A Skinned Multi-Person Linear Model. ACM transactions on graphics (TOG) 34, 6 (2015), 1–16. Google ScholarDigital Library
28. Mickaël Ly, Romain Casati, Florence Bertails-Descoubes, Mélina Skouras, and Laurence Boissieux. 2018. Inverse Elastic Shell Design with Contact and Friction. ACM Transaction on Graphics (TOG) 37, 6 (2018), 1–16. Google ScholarDigital Library
29. Qianli Ma, Jinlong Yang, Anurag Ranjan, Sergi Pujades, Gerard Pons-Moll, Siyu Tang, and Michael J Black. 2020. Learning to Dress 3D People in Generative Clothing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, Virtual, 6469–6478. https://cape.is.tue.mpg.de.Google ScholarCross Ref
30. Qianli Ma, Jinlong Yang, Siyu Tang, and Michael J Black. 2021. The Power of Points for Modeling Humans in Clothing. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, Virtual, 10974–10984.Google ScholarCross Ref
31. Andre Martins and Ramon Astudillo. 2016. From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification. In Proceedings of The 33rd International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 48), Maria Florina Balcan and Kilian Q. Weinberger (Eds.). PMLR, New York, New York, USA, 1614–1623. https://proceedings.mlr.press/v48/martins16.htmlGoogle Scholar
32. Yuwei Meng, Charlie C.L. Wang, and Xiaogang Jin. 2012. Flexible shape control for automatic resizing of apparel products. CAD Computer Aided Design 44, 1 (2012), 68–76. Google ScholarDigital Library
33. Kaichun Mo, Paul Guerrero, Li Yi, Hao Su, Peter Wonka, Niloy Mitra, and Leonidas J. Guibas. 2019. StructureNet: Hierarchical Graph Networks for 3D Shape Generation. ACM Transactions on Graphics (TOG) 38, 6 (8 2019), 1–19. http://arxiv.org/abs/1908.00575Google ScholarDigital Library
34. Juan Montes, Bernhard Thomaszewski, Sudhir Mudur, and Tiberiu Popa. 2020. Computational Design of Skintight Clothing. ACM Trans. Graph 39, 4 (2020), 12. Google ScholarDigital Library
35. Vacit Oguz Yazici, Abel Gonzalez-Garcia, Arnau Ramisa, Bartlomiej Twardowski, and Joost van de Weijer. 2020. Orderless Recurrent Models for Multi-label Classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, Virtual, 13440–13449.Google Scholar
36. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems 32, H. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. Fox, and R. Garnett (Eds.). Curran Associates, Inc., 8024–8035. http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdfGoogle Scholar
37. Chaitanya Patel, Zhouyingcheng Liao, and Gerard Pons-Moll. 2020. TailorNet: Predicting Clothing in 3D as a Function of Human Pose, Shape and Garment Style. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, Virtual, 7365–7375. http://arxiv.org/abs/2003.04583Google ScholarCross Ref
38. Dominic Roberts, Ara Danielyan, Hang Chu, Mani Golparvar-Fard, and David Forsyth. 2021. LSD-StructureNet: Modeling Levels of Structural Detail in 3D Part Hierarchies. In Proceedings of the IEEE/CVF International Conference on Computer Vision. IEEE, Virtual, 5836–5845. http://arxiv.org/abs/2108.13459Google ScholarCross Ref
39. Igor Santesteban, Miguel A. Otaduy, and Dan Casas. 2019. Learning-Based Animation of Clothing for Virtual Try-On. Computer Graphics Forum 38, 2 (2019), 355–366. Google ScholarCross Ref
40. Igor Santesteban, Nils Thuerey, Miguel A Otaduy, and Dan Casas. 2021. Self-Supervised Collision Handling via Generative 3D Garment Models for Virtual Try-On. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 11763–11773.Google ScholarCross Ref
41. Matthew Schultz and Thorsten Joachims. 2003. Learning a Distance Metric from Relative Comparisons. Advances in neural information processing systems 16 (2003), 41–48.Google Scholar
42. Nicholas Sharp and Keenan Crane. 2018. Variational surface cutting. ACM Transactions on Graphics 37, 4 (2018), 1–13. Google ScholarDigital Library
43. Yu Shen, Junbang Liang, and Ming C. Lin. 2020. GAN-Based Garment Generation Using Sewing Pattern Images. In Computer Vision-ECCV 2020: 16th European Conference, Vol. 16. Springer International Publishing, Glasgow, UK, 225–247. Google ScholarCross Ref
44. Leslie N. Smith. 2018. A Disciplined Approach to Neural Network Hyper-Parameters: PART 1 – Learning Rate, Batch Size, Momentum, and Weight Decay. https://github.com/lnsmith54/hyperParam1.Google Scholar
45. Zhaoqi Su, Tao Yu, Yangang Wang, Yipeng Li, and Yebin Liu. 2020. DeepCloth : Neural Garment Representation for Shape and Style Editing. arXiv preprint arXiv:2011.14619 (2020), 1–10.Google Scholar
46. Garvita Tiwari, Bharat Lal Bhatnagar, Tony Tung, and Gerard Pons-Moll. 2020. SIZER: A Dataset and Model for Parsing 3D Clothing and Learning Size Sensitive 3D Clothing. http://arxiv.org/abs/2007.11610Google Scholar
47. Charlie C.L. Wang, Yu Wang, and Matthew M.F. Yuen. 2003. Feature based 3D garment design through 2D sketches. CAD Computer Aided Design 35, 7 (6 2003), 659–672. Google ScholarCross Ref
48. Charlie C.L. Wang, Yu Wang, and Matthew M.F. Yuen. 2005. Design automation for customized apparel products. CAD Computer Aided Design 37, 7 (6 2005), 675–691. Google ScholarDigital Library
49. Huamin Wang. 2018. Rule-free sewing pattern adjustment with precision and efficiency. ACM Transactions on Graphics 37, 4 (2018), 1–13. Google ScholarDigital Library
50. Jin Wang, Guodong Lu, Weilong Li, Long Chen, and Yoshiyuki Sakaguti. 2009. Interactive 3D garment design with constrained contour curves and style curves. CAD Computer Aided Design 41, 9 (9 2009), 614–625. Google ScholarDigital Library
51. Kai Wang, Paul Guerrero, Vladimir Kim, Siddhartha Chaudhuri, Minhyuk Sung, and Daniel Ritchie. 2021. The Shape Part Slot Machine: Contact-based Reasoning for Generating 3D Shapes from Parts. arXiv preprint arXiv:2112.00584 (2021), 1–19. http://arxiv.org/abs/2112.00584Google Scholar
52. Tuanfeng Y. Wang, Duygu Ceylan, Jovan Popović, and Niloy J. Mitra. 2018a. Learning a shared shape space for multimodal garment design. ACM Transactions on Graphics 37, 6 (12 2018), 1–13. Google ScholarDigital Library
53. Tuanfeng Y Wang and MiHoYo Inc. 2019. Learning an Intrinsic Garment Space for Interactive Authoring of Garment Animation. ACM Transactions on Graphics (TOG) 38, 6 (2019), 1–12. Google ScholarDigital Library
54. Yizhi Wang and Zhouhui Lian. 2021. DeepVecFont: Synthesizing High-quality Vector Fonts via Dual-modality Learning. ACM Transactions on Graphics (TOG) 40, 6 (2021), 1–15. http://arxiv.org/abs/2110.06688Google ScholarDigital Library
55. Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E. Sarma, Michael M. Bronstein, and Justin M. Solomon. 2018b. Dynamic Graph CNN for Learning on Point Clouds. ACM Transactions on Graphics 38, 5 (1 2018), Article 146. http://arxiv.org/abs/1801.07829Google Scholar
56. Katja Wolff, Philipp Herholz, Verena Ziegler, Frauke Link, Nico Brügel, and Olga Sorkine-Hornung. 2021. 3D Custom Fit Garment Design with Body Movement. arXiv preprint arXiv:2102.05462 (2021), 1–12. http://arxiv.org/abs/2102.05462Google Scholar
57. Zhijie Wu, Xiang Wang, Di Lin, Dani Lischinski, Daniel Cohen-Or, and Hui Huang. 2019. SagNet: Structure-aware generative network for 3D-shape modeling. ACM Transactions on Graphics 38, 4 (7 2019), 1–14. Google ScholarDigital Library
58. Shan Yang, Zherong Pan, Tanya Amert, Ke Wang, Licheng Yu, Tamara Berg, and Ming C. Lin. 2018. Physics-inspired garment recovery from a single-view image. ACM Transactions on Graphics 37, 5 (11 2018), 1–14. Google ScholarDigital Library
59. Yang Yunchu and Zhang Weiyuan. 2007. Prototype garment pattern flattening based on individual 3D virtual dummy. International Journal of Clothing Science and Technology 19, 5 (2007), 334–348. Google ScholarCross Ref
60. Ilya Zakharkin, Kirill Mazur, Artur Grigorev, and Victor Lempitsky. 2021. Point-Based Modeling of Human Clothing. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, Virtual, 14718–14727. http://arxiv.org/abs/2104.08230Google ScholarCross Ref
61. Heming Zhu, Yu Cao, Hang Jin, Weikai Chen, Dong Du, Zhangye Wang, Shuguang Cui, and Xiaoguang Han. 2020. Deep Fashion3D: A Dataset and Benchmark for 3D Garment Reconstruction from Single Images. http://arxiv.org/abs/2003.12753https://kv2000.github.io/2020/03/25/deepFashion3DRevisited/Google Scholar