“Learning to predict part mobility from a single static snapshot”
Conference:
Type(s):
Title:
- Learning to predict part mobility from a single static snapshot
Session/Category Title: Learning Geometry
Presenter(s)/Author(s):
Abstract:
We introduce a method for learning a model for the mobility of parts in 3D objects. Our method allows not only to understand the dynamic functionalities of one or more parts in a 3D object, but also to apply the mobility functions to static 3D models. Specifically, the learned part mobility model can predict mobilities for parts of a 3D object given in the form of a single static snapshot reflecting the spatial configuration of the object parts in 3D space, and transfer the mobility from relevant units in the training data. The training data consists of a set of mobility units of different motion types. Each unit is composed of a pair of 3D object parts (one moving and one reference part), along with usage examples consisting of a few snapshots capturing different motion states of the unit. Taking advantage of a linearity characteristic exhibited by most part motions in everyday objects, and utilizing a set of part-relation descriptors, we define a mapping from static snapshots to dynamic units. This mapping employs a motion-dependent snapshot-to-unit distance obtained via metric learning. We show that our learning scheme leads to accurate motion prediction from single static snapshots and allows proper motion transfer. We also demonstrate other applications such as motion-driven object detection and motion hierarchy construction.
References:
1. Noa Fish, Melinos Averkiou, Oliver van Kaick, Olga Sorkine-Hornung, Daniel Cohen-Or, and Niloy J. Mitra. 2014. Meta-representation of Shape Families. ACM Trans. on Graphics 33, 4 (2014), 34:1–11.
2. Brendan J. Frey and Delbert Dueck. 2007. Clustering by Passing Messages Between Data Points. Science 315 (2007), 972–976. Cross Ref
3. Natasha Gelfand and Leonidas J. Guibas. 2004. Shape Segmentation Using Local Slippage Analysis. In Proc. SGP. 214–223.
4. H. Grabner, J. Gall, and L. Van Gool. 2011. What makes a chair a chair?. In Proc. IEEE Conf. on Computer Vision & Pattern Recognition. IEEE, 1529–1536.
5. Paul Guerrero, Niloy J. Mitra, and Peter Wonka. 2016. RAID: A Relation-augmented Image Descriptor. ACM Trans. on Graphics 35, 4 (2016), 46:1–12.
6. Jianwei Guo, Dong-Ming Yan, Er Lid, Weiming Dong, Peter Wonka, and Xiaopeng Zhang. 2013. Illustrating the disassembly of 3D models. Computers & Graphics 37, 6 (2013), 574–581.
7. T. Hermans, F. Li, J. M. Rehg, and A. F. Bobick. 2013. Learning contact locations for pushing and orienting unknown objects. In Int. Conf. on Humanoid Robots. IEEE, 435–442.
8. Ruizhen Hu, Oliver van Kaick, Bojian Wu, Hui Huang, Ariel Shamir, and Hao Zhang. 2016. Learning How Objects Function via Co-Analysis of Interactions. ACM Trans. on Graphics 35, 4 (2016), 47:1–12.
9. Ruizhen Hu, Chenyang Zhu, Oliver van Kaick, Ligang Liu, Ariel Shamir, and Hao Zhang. 2015. Interaction Context (ICON): Towards a Geometric Functionality Descriptor. ACM Trans. on Graphics 34, 4 (2015), 83:1–12.
10. Vladimir G. Kim, Siddhartha Chaudhuri, Leonidas Guibas, and Thomas Funkhouser. 2014. Shape2Pose: Human-Centric Shape Analysis. ACM Trans. on Graphics 33, 4 (2014), 120:1–12.
11. Paul G. Kry and Dinesh K. Pai. 2006. Interaction Capture and Synthesis. ACM Trans. on Graphics 25, 3 (2006), 872–880.
12. Hao Li, Guowei Wan, Honghua Li, Andrei Sharf, Kai Xu, and Baoquan Chen. 2016. Mobility Fitting using 4D RANSAC. Computer Graphics Forum 35, 5 (2016), 79–88. Cross Ref
13. Niloy J. Mitra, Yong-Liang Yang, Dong-Ming Yan, Wilmot Li, and Maneesh Agrawala. 2010. Illustrating How Mechanical Assemblies Work. ACM Trans. on Graphics 29, 4 (2010), 58:1–12.
14. Luca Del Pero, Susanna Ricco, Rahul Sukthankar, and Vittorio Ferrari. 2016. Discovering the physical parts of an articulated object class from multiple videos. In Proc. IEEE Conf. on Computer Vision & Pattern Recognition. IEEE.
15. Sören Pirk, Vojtech Krs, Kaimo Hu, Suren Deepak Rajasekaran, Hao Kang, Bedrich Benes, Yusuke Yoshiyasu, and Leonidas J. Guibas. 2017. Understanding and Exploiting Object Interaction Landscapes. ACM Trans. on Graphics 36, 3 (2017), 31:1–14.
16. Manolis Savva, Angel X. Chang, Pat Hanrahan, Matthew Fisher, and Matthias Nießner. 2014. SceneGrok: Inferring Action Maps in 3D Environments. ACM Trans. on Graphics 33, 6 (2014), 212:1–10.
17. Manolis Savva, Angel X. Chang, Pat Hanrahan, Matthew Fisher, and Matthias Nießner. 2016. PiGraphs: Learning Interaction Snapshots from Observations. ACM Trans. on Graphics 35, 4 (2016), 139:1–12.
18. Matthew Schultz and Thorsten Joachims. 2003. Learning a distance metric from relative comparisons. Advances in neural information processing systems (NIPS) 16 (2003).
19. Tianjia Shao, Wilmot Li, Kun Zhou, Weiwei Xu, Baining Guo, and Niloy J. Mitra. 2013. Interpreting Concept Sketches. ACM Trans. on Graphics 32, 4 (2013), 56:1–10.
20. A. Sharf, H. Huang, C. Liang, J. Zhang, B. Chen, and M. Gong. 2013. Mobility-Trees for Indoor Scenes Manipulation. Computer Graphics Forum 33, 1 (2013), 2–14.
21. Harald Steck. 2007. Hinge rank loss and the area under the ROC curve. In European Conference on Machine Learning. Springer, 347–358.
22. Jörg Stückler, Benedikt Waldvogel, Hannes Schulz, and Sven Behnke. 2015. Dense Real-time Mapping of Object-class Semantics from RGB-D Video. J. Real-Time Image Process. 10, 4 (2015), 599–609.
23. Art Tevs, Alexander Berner, Michael Wand, Ivo Ihrke, Martin Bokeloh, Jens Kerber, and Hans-Peter Seidel. 2012. Animation Cartography – Intrinsic Reconstruction of Shape and Motion. ACM Trans. on Graphics 31, 2 (2012), 12:1–15.
24. P. Wei, Y. Zhao, N. Zheng, and S. C. Zhu. 2017. Modeling 4D Human-Object Interactions for Joint Event Segmentation, Recognition, and Object Localization. IEEE Trans. Pattern Analysis & Machine Intelligence 39, 6 (2017), 1165–1179.
25. Weiwei Xu, Jun Wang, KangKang Yin, Kun Zhou, Michiel van de Panne, Falai Chen, and Baining Guo. 2009. Joint-aware Manipulation of Deformable Models. ACM Trans. on Graphics 28, 3 (2009), 35:1–9.
26. Tianfan Xue, Jiajun Wu, Katherine L Bouman, and William T Freeman. 2016. Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks. In Advances in neural information processing systems (NIPS).
27. Xi Zhao, He Wang, and Taku Komura. 2014. Indexing 3D Scenes Using the Interaction Bisector Surface. ACM Trans. on Graphics 33, 3 (2014), 22:1–14.
28. Y. Zhu, Y. Zhao, and S. C. Zhu. 2015. Understanding tools: Task-oriented object modeling, learning and recognition. In Proc. IEEE Conf. on Computer Vision & Pattern Recognition. IEEE, 2855–2864.
