“ReparamCAD: Zero-shot CAD Re-Parameterization for Interactive Manipulation” by Kodnongbua, Jones, Ahmad, Kim and Schulz
Conference:
Type(s):
Title:
- ReparamCAD: Zero-shot CAD Re-Parameterization for Interactive Manipulation
Session/Category Title: Navigating Shape Spaces
Presenter(s)/Author(s):
Abstract:
Parametric CAD models encode entire families of shapes that should, in principle, be easy for designers to explore, but are in practice difficult to manipulate due to implicit semantic constraints between parameter values. Finding and enforcing these semantic constraints purely from geometry or programmatic shape representations is not possible because these constraints are ultimately a form of design intent, and are informed by the designer’s experience and semantic in the real world. While individual pieces of geometry cannot encode this contextual knowledge, is it encoded in large foundation models. We use large language and image models to understand what the meaningful space of variation is for a shape, and synthesize a new parametric CAD program that captures these variations, allowing for easy design space exploration along meaningful design axes.
References:
[1]
Panos Achlioptas, Ian Huang, Minhyuk Sung, Sergey Tulyakov, and Leonidas Guibas. 2022. ChangeIt3D: Language-assisted 3d shape edits and deformations.
[2]
Maaz Bin Safeer Ahmad, Jonathan Ragan-Kelley, Alvin Cheung, and Shoaib Kamil. 2019. Automatically Translating Image Processing Libraries to Halide. ACM Trans. Graph. 38, 6, Article 204 (nov 2019), 13 pages. https://doi.org/10.1145/3355089.3356549
[3]
Christian Bessiere, Remi Coletta, Eugene C. Freuder, and Barry O’Sullivan. 2004. Leveraging the Learning Power of Examples in Automated Constraint Acquisition. In Principles and Practice of Constraint Programming – CP 2004, Mark Wallace (Ed.). Springer Berlin Heidelberg, Berlin, Heidelberg, 123–137.
[4]
Martin Bokeloh, Michael Wand, Hans-Peter Seidel, and Vladlen Koltun. 2012. An algebraic model for parameterized shape editing. ACM Transactions on Graphics (TOG) 31, 4 (2012), 1–10.
[5]
José Cambronero, Sumit Gulwani, Vu Le, Daniel Perelman, Arjun Radhakrishna, Clint Simon, and Ashish Tiwari. 2023. FlashFill++: Scaling Programming by Example by Cutting to the Chase. In Principles of Programming Languages. ACM SIGPLAN, ACM. https://www.microsoft.com/en-us/research/publication/flashfill-scaling-programming-by-example-by-cutting-to-the-chase/
[6]
David Cao, Rose Kunkel, Chandrakana Nandi, Max Willsey, Zachary Tatlock, and Nadia Polikarpova. 2023. Babble: Learning Better Abstractions with E-Graphs and Anti-Unification. Proc. ACM Program. Lang. 7, POPL, Article 14 (jan 2023), 29 pages. https://doi.org/10.1145/3571207
[7]
Dan Cascaval, Mira Shalah, Phillip Quinn, Rastislav Bodík, Maneesh Agrawala, and Adriana Schulz. 2021. Differentiable 3D CAD Programs for Bidirectional Editing. CoRR abs/2110.01182 (2021). arXiv:2110.01182https://arxiv.org/abs/2110.01182
[8]
Siddhartha Chaudhuri, Evangelos Kalogerakis, Stephen Giguere, and Thomas Funkhouser. 2013. Attribit: Content Creation with Semantic Attributes. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (St. Andrews, Scotland, United Kingdom) (UIST ’13). Association for Computing Machinery, New York, NY, USA, 193–202. https://doi.org/10.1145/2501988.2502008
[9]
Kevin Chen, Christopher B. Choy, Manolis Savva, Angel X. Chang, Thomas Funkhouser, and Silvio Savarese. 2019. Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddings. In Computer Vision – ACCV 2018, C. V. Jawahar, Hongdong Li, Greg Mori, and Konrad Schindler (Eds.). Springer International Publishing, Cham, 100–116.
[10]
Zhiqin Chen and Hao Zhang. 2019. Learning implicit fields for generative shape modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5939–5948.
[11]
Gianmarco Cherchi, Fabio Pellacini, Marco Attene, and Marco Livesu. 2022. Interactive and Robust Mesh Booleans. ACM Transactions on Graphics (SIGGRAPH Asia 2022) 41, 6 (2022). https://doi.org/10.1145/3550454.3555460
[12]
Tao Du, Jeevana Priya Inala, Yewen Pu, Andrew Spielberg, Adriana Schulz, Daniela Rus, Armando Solar-Lezama, and Wojciech Matusik. 2018. InverseCSG: Automatic Conversion of 3D Models to CSG Trees. ACM Transactions on Graphics 37, 6, Article 213 (dec 2018), 16 pages. https://doi.org/10.1145/3272127.3275006
[13]
Kevin Ellis, Maxwell Nye, Yewen Pu, Felix Sosa, Joshua B. Tenenbaum, and Armando Solar-Lezama. 2019. Write, Execute, Assess: Program Synthesis with a REPL. Curran Associates Inc., Red Hook, NY, USA.
[14]
Kevin Ellis, Catherine Wong, Maxwell Nye, Mathias Sablé-Meyer, Lucas Morales, Luke Hewitt, Luc Cary, Armando Solar-Lezama, and Joshua B. Tenenbaum. 2021. DreamCoder: Bootstrapping Inductive Program Synthesis with Wake-Sleep Library Learning. In Proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation (Virtual, Canada) (PLDI 2021). Association for Computing Machinery, New York, NY, USA, 835–850. https://doi.org/10.1145/3453483.3454080
[15]
Adejuyigbe O. Fajemisin, Donato Maragno, and Dick den Hertog. 2023. Optimization with constraint learning: A framework and survey. European Journal of Operational Research (2023). https://doi.org/10.1016/j.ejor.2023.04.041
[16]
Grigory Fedyukovich and Rastislav Bodík. 2018. Accelerating Syntax-Guided Invariant Synthesis. In Tools and Algorithms for the Construction and Analysis of Systems, Dirk Beyer and Marieke Huisman (Eds.). Springer International Publishing, Cham, 251–269.
[17]
Noa Fish, Melinos Averkiou, Oliver Van Kaick, Olga Sorkine-Hornung, Daniel Cohen-Or, and Niloy J Mitra. 2014. Meta-representation of shape families. ACM Transactions on Graphics (TOG) 33, 4 (2014), 1–11.
[18]
Rao Fu, Xiao Zhan, Yiwen Chen, Daniel Ritchie, and Srinath Sridhar. 2023. ShapeCrafter: A Recursive Text-Conditioned 3D Shape Generation Model. arxiv:2207.09446 [cs.CV]
[19]
Mathieu Gaillard, Vojtěch Krs, Giorgio Gori, Radomír Měch, and Bedrich Benes. 2022. Automatic differentiable procedural modeling. In Computer Graphics Forum, Vol. 41. Wiley Online Library, 289–307.
[20]
Ran Gal, Olga Sorkine, Niloy Mitra, and Daniel Cohen-Or. 2009a. iWires: An Analyze-and-Edit Approach to Shape Manipulation. ACM Transactions on Graphics (proceedings of ACM SIGGRAPH) 28, 3 (2009), 33:1–33:10.
[21]
Ran Gal, Olga Sorkine, Niloy J Mitra, and Daniel Cohen-Or. 2009b. iWIRES: An analyze-and-edit approach to shape manipulation. In ACM SIGGRAPH 2009 papers. 1–10.
[22]
Zekun Hao, Hadar Averbuch-Elor, Noah Snavely, and Serge Belongie. 2020. Dualsdf: Semantic shape manipulation using a two-level representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 7631–7641.
[23]
Amir Hertz, Or Perel, Raja Giryes, Olga Sorkine-Hornung, and Daniel Cohen-Or. 2022. Spaghetti: Editing implicit shapes through part aware generation. ACM Transactions on Graphics (TOG) 41, 4 (2022), 1–20.
[24]
Takeo Igarashi, Tomer Moscovich, and John F Hughes. 2005. As-rigid-as-possible shape manipulation. ACM transactions on Graphics (TOG) 24, 3 (2005), 1134–1141.
[25]
Alec Jacobson, Ilya Baran, Jovan Popovic, and Olga Sorkine. 2011. Bounded biharmonic weights for real-time deformation.ACM Trans. Graph. 30, 4 (2011), 78.
[26]
Ajay Jain, Ben Mildenhall, Jonathan T. Barron, Pieter Abbeel, and Ben Poole. 2022. Zero-Shot Text-Guided Object Generation with Dream Fields. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 857–866. https://doi.org/10.1109/CVPR52688.2022.00094
[27]
Chiyu Jiang, Jingwei Huang, Andrea Tagliasacchi, and Leonidas J Guibas. 2020. Shapeflow: Learnable deformation flows among 3d shapes. Advances in Neural Information Processing Systems 33 (2020), 9745–9757.
[28]
Benjamin T Jones, Michael Hu, Milin Kodnongbua, Vladimir G Kim, and Adriana Schulz. 2023b. Self-Supervised Representation Learning for CAD. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 21327–21336.
[29]
R Kenny Jones, Theresa Barton, Xianghao Xu, Kai Wang, Ellen Jiang, Paul Guerrero, Niloy J Mitra, and Daniel Ritchie. 2020. Shapeassembly: Learning to generate programs for 3d shape structure synthesis. ACM Transactions on Graphics (TOG) 39, 6 (2020), 1–20.
[30]
R Kenny Jones, David Charatan, Paul Guerrero, Niloy J Mitra, and Daniel Ritchie. 2021. ShapeMOD: macro operation discovery for 3D shape programs. ACM Transactions on Graphics (TOG) 40, 4 (2021), 1–16.
[31]
R Kenny Jones, Paul Guerrero, Niloy J Mitra, and Daniel Ritchie. 2023a. ShapeCoder: Discovering Abstractions for Visual Programs from Unstructured Primitives. arXiv preprint arXiv:2305.05661 (2023).
[32]
R Kenny Jones, Aalia Habib, Rana Hanocka, and Daniel Ritchie. 2022. The neurally-guided shape parser: Grammar-based labeling of 3d shape regions with approximate inference. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 11614–11623.
[33]
Nasir Mohammad Khalid, Tianhao Xie, Eugene Belilovsky, and Tiberiu Popa. 2022. CLIP-Mesh: Generating textured meshes from text using pretrained image-text models. In SIGGRAPH Asia 2022 Conference Papers. ACM. https://doi.org/10.1145/3550469.3555392
[34]
R. Killick, P. Fearnhead, and I. A. Eckley. 2012. Optimal Detection of Changepoints With a Linear Computational Cost. J. Amer. Statist. Assoc. 107, 500 (2012), 1590–1598. https://doi.org/10.1080/01621459.2012.737745 arXiv:https://doi.org/10.1080/01621459.2012.737745
[35]
Samuli Laine, Janne Hellsten, Tero Karras, Yeongho Seol, Jaakko Lehtinen, and Timo Aila. 2020. Modular Primitives for High-Performance Differentiable Rendering. ACM Transactions on Graphics 39, 6 (2020).
[36]
Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin. 2023. Magic3D: High-Resolution Text-to-3D Content Creation. arxiv:2211.10440 [cs.CV]
[37]
Minghua Liu, Minhyuk Sung, Radomir Mech, and Hao Su. 2021. Deepmetahandles: Learning deformation meta-handles of 3d meshes with biharmonic coordinates. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 12–21.
[38]
Zhengzhe Liu, Yi Wang, Xiaojuan Qi, and Chi-Wing Fu. 2022. Towards Implicit Text-Guided 3D Shape Generation. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 17875–17885. https://doi.org/10.1109/CVPR52688.2022.01737
[39]
Elie Michel and Tamy Boubekeur. 2021. DAG amendment for inverse control of parametric shapes. ACM Transactions on Graphics (TOG) 40, 4 (2021), 1–14.
[40]
Niloy J Mitra, Michael Wand, Hao Zhang, Daniel Cohen-Or, Vladimir Kim, and Qi-Xing Huang. 2014. Structure-aware shape processing. In ACM SIGGRAPH 2014 Courses. 1–21.
[41]
Paritosh Mittal, Yen-Chi Cheng, Maneesh Singh, and Shubham Tulsiani. 2023. AutoSDF: Shape Priors for 3D Completion, Reconstruction and Generation. arxiv:2203.09516 [cs.CV]
[42]
Kaichun Mo, Paul Guerrero, Li Yi, Hao Su, Peter Wonka, Niloy Mitra, and Leonidas Guibas. 2019. StructureNet: Hierarchical Graph Networks for 3D Shape Generation. ACM Transactions on Graphics (TOG), Siggraph Asia 2019 38, 6 (2019), Article 242.
[43]
Chandrakana Nandi, James R Wilcox, Pavel Panchekha, Taylor Blau, Dan Grossman, and Zachary Tatlock. 2018. Functional programming for compiling and decompiling computer-aided design. Proceedings of the ACM on Programming Languages 2, ICFP (2018), 1–31.
[44]
Chandrakana Nandi, Max Willsey, Adam Anderson, James R. Wilcox, Eva Darulova, Dan Grossman, and Zachary Tatlock. 2020. Synthesizing Structured CAD Models with Equality Saturation and Inverse Transformations. In Proceedings of the 41st ACM SIGPLAN Conference on Programming Language Design and Implementation (London, UK) (PLDI 2020). Association for Computing Machinery, New York, NY, USA, 31–44. https://doi.org/10.1145/3385412.3386012
[45]
Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. 2022b. Point-E: A System for Generating 3D Point Clouds from Complex Prompts. arxiv:2212.08751 [cs.CV]
[46]
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya Sutskever, and Mark Chen. 2022a. GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models. In Proceedings of the 39th International Conference on Machine Learning(Proceedings of Machine Learning Research, Vol. 162), Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (Eds.). PMLR, 16784–16804. https://proceedings.mlr.press/v162/nichol22a.html
[47]
Maks Ovsjanikov, Wilmot Li, Leonidas Guibas, and Niloy J Mitra. 2011. Exploration of continuous variability in collections of 3d shapes. ACM Transactions on Graphics (TOG) 30, 4 (2011), 1–10.
[48]
Ofek Pearl, Itai Lang, Yuhua Hu, Raymond A Yeh, and Rana Hanocka. 2022. GeoCode: Interpretable Shape Programs. arXiv preprint arXiv:2212.11715 (2022).
[49]
Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. 2022. DreamFusion: Text-to-3D using 2D Diffusion. arxiv:2209.14988 [cs.CV]
[50]
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings of the 38th International Conference on Machine Learning(Proceedings of Machine Learning Research, Vol. 139), Marina Meila and Tong Zhang (Eds.). PMLR, 8748–8763. https://proceedings.mlr.press/v139/radford21a.html
[51]
Daniel Ritchie, Paul Guerrero, R. Kenny Jones, Niloy J. Mitra, Adriana Schulz, Karl D. D. Willis, and Jiajun Wu. 2023. Neurosymbolic Models for Computer Graphics. Computer Graphics Forum (2023). https://doi.org/10.1111/cgf.14775
[52]
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-Resolution Image Synthesis with Latent Diffusion Models. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 10674–10685. https://doi.org/10.1109/CVPR52688.2022.01042
[53]
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. 2022. Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding. In Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.). Vol. 35. Curran Associates, Inc., 36479–36494. https://proceedings.neurips.cc/paper_files/paper/2022/file/ec795aeadae0b7d230fa35cbaf04c041-Paper-Conference.pdf
[54]
Aditya Sanghi, Hang Chu, Joseph G. Lambourne, Ye Wang, Chin-Yi Cheng, Marco Fumero, and Kamal Rahimi Malekshan. 2022. CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 18582–18592. https://doi.org/10.1109/CVPR52688.2022.01805
[55]
Aditya Sanghi, Rao Fu, Vivian Liu, Karl Willis, Hooman Shayani, Amir Hosein Khasahmadi, Srinath Sridhar, and Daniel Ritchie. 2023. CLIP-Sculptor: Zero-Shot Generation of High-Fidelity and Diverse Shapes from Natural Language. arxiv:2211.01427 [cs.CV]
[56]
Junyoung Seo, Wooseok Jang, Min-Seop Kwak, Jaehoon Ko, Hyeonsu Kim, Junho Kim, Jin-Hwa Kim, Jiyoung Lee, and Seungryong Kim. 2023. Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation. arxiv:2303.07937 [cs.CV]
[57]
Olga Sorkine and Mario Botsch. 2009. Interactive Shape Modeling and Deformation. In Eurographics (Tutorials). 11–37.
[58]
Olga Sorkine, Daniel Cohen-Or, Yaron Lipman, Marc Alexa, Christian Rössl, and H-P Seidel. 2004. Laplacian surface editing. In Proceedings of the 2004 Eurographics/ACM SIGGRAPH symposium on Geometry processing. 175–184.
[59]
Robert W Sumner, Johannes Schmid, and Mark Pauly. 2007. Embedded deformation for shape manipulation. In ACM siggraph 2007 papers. 80–es.
[60]
Minhyuk Sung, Zhenyu Jiang, Panos Achlioptas, Niloy J. Mitra, and Leonidas J. Guibas. 2020. DeformSyncNet: Deformation Transfer via Synchronized Shape Deformation Spaces. ACM Trans. Graph. 39, 6, Article 261 (nov 2020), 16 pages. https://doi.org/10.1145/3414685.3417783
[61]
Shubham Tulsiani, Hao Su, Leonidas J Guibas, Alexei A Efros, and Jitendra Malik. 2017. Learning shape abstractions by assembling volumetric primitives. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2635–2643.
[62]
Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. 2018. Pixel2mesh: Generating 3d mesh models from single rgb images. In Proceedings of the European conference on computer vision (ECCV). 52–67.
[63]
Weiyue Wang, Duygu Ceylan, Radomir Mech, and Ulrich Neumann. 2019. 3dn: 3d deformation network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1038–1046.
[64]
Fangyin Wei, Elena Sizikova, Avneesh Sud, Szymon Rusinkiewicz, and Thomas Funkhouser. 2020. Learning to infer semantic parameters for 3D shape editing. In 2020 International Conference on 3D Vision (3DV). IEEE, 434–442.
[65]
Jiale Xu, Xintao Wang, Weihao Cheng, Yan-Pei Cao, Ying Shan, Xiaohu Qie, and Shenghua Gao. 2023. Dream3D: Zero-Shot Text-to-3D Synthesis Using 3D Shape Prior and Text-to-Image Diffusion Models. arxiv:2212.14704 [cs.CV]
[66]
Evan Yares. 2013. The failed promise of parametric CAD part 1: From the beginning. https://www.3dcadworld.com/the-failed-promise-of-parametric-cad/. (Accessed on 09/06/2019).
[67]
Wang Yifan, Noam Aigerman, Vladimir G Kim, Siddhartha Chaudhuri, and Olga Sorkine-Hornung. 2020. Neural cages for detail-preserving 3d deformations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 75–83.
[68]
Kangxue Yin, Zhiqin Chen, Siddhartha Chaudhuri, Matthew Fisher, Vladimir G Kim, and Hao Zhang. 2020. Coalesce: Component assembly by learning to synthesize connections. In 2020 International Conference on 3D Vision (3DV). IEEE, 61–70.
[69]
Fenggen Yu, Zhiqin Chen, Manyi Li, Aditya Sanghi, Hooman Shayani, Ali Mahdavi-Amiri, and Hao Zhang. 2022. CAPRI-Net: Learning Compact CAD Shapes With Adaptive Primitive Assembly. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 11768–11778.
[70]
Mehmet Ersin Yumer, Siddhartha Chaudhuri, Jessica K Hodgins, and Levent Burak Kara. 2015. Semantic shape editing using deformation handles. ACM Transactions on Graphics (TOG) 34, 4 (2015), 1–12.
[71]
xiaohui zeng, Arash Vahdat, Francis Williams, Zan Gojcic, Or Litany, Sanja Fidler, and Karsten Kreis. 2022. LION: Latent Point Diffusion Models for 3D Shape Generation. In Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.). Vol. 35. Curran Associates, Inc., 10021–10039. https://proceedings.neurips.cc/paper_files/paper/2022/file/40e56dabe12095a5fc44a6e4c3835948-Paper-Conference.pdf
[72]
X Zheng, Yang Liu, P Wang, and Xin Tong. 2022. SDF-StyleGAN: Implicit SDF-Based StyleGAN for 3D Shape Generation. In Computer Graphics Forum, Vol. 41. Wiley Online Library, 52–63.
