“Relationship templates for creating scene variations”
Conference:
Type(s):
Title:
- Relationship templates for creating scene variations
Session/Category Title: Shape Semantics
Presenter(s)/Author(s):
Abstract:
We propose a novel example-based approach to synthesize scenes with complex relations, e.g., when one object is ‘hooked’, ‘surrounded’, ‘contained’ or ‘tucked into’ another object. Existing relationship descriptors used in automatic scene synthesis methods are based on contacts or relative vectors connecting the object centers. Such descriptors do not fully capture the geometry of spatial interactions, and therefore cannot describe complex relationships. Our idea is to enrich the description of spatial relations between object surfaces by encoding the geometry of the open space around objects, and use this as a template for fitting novel objects. To this end, we introduce relationship templates as descriptors of complex relationships; they are computed from an example scene and combine the interaction bisector surface (IBS) with a novel feature called the space coverage feature (SCF), which encodes the open space in the frequency domain. New variations of a scene can be synthesized efficiently by fitting novel objects to the template. Our method greatly enhances existing automatic scene synthesis approaches by allowing them to handle complex relationships, as validated by our user studies. The proposed method generalizes well, as it can form complex relationships with objects that have a topology and geometry very different from the example scene.
References:
1. Barber, C. B., Dobkin, D. P., and Huhdanpaa, H. 1996. The quickhull algorithm for convex hulls. ACM Trans. Math. Soft. 22, 4, 469483.
2. Chen, K., Lai, Y.-K., Wu, Y.-X., Martin, R., and Hu, S.-M. 2014. Automatic semantic modeling of indoor scenes from low-quality rgb-d data using contextual information. ACM TOG 33, 6.
3. Coyne, B., and Sproat, R. 2001. Wordseye: An automatic text-to-scene conversion system. In ACM SIGGRAPH, ACM, New York, NY, USA, 487–496.
4. Coyne, B., Sproat, R., and Hirschberg, J. 2010. Spatial relations in text-to-scene conversion. In Computational Models of Spatial Language Interpretation, at Spatial Cognition 2010.
5. Fei-Fei, L., and Perona, P. 2005. A bayesian hierarchical model for learning natural scene categories. In IEEE CVPR, vol. 2, IEEE, 524–531.
6. Fisher, M., and Hanrahan, P. 2010. Context-based search for 3d models. In ACM SIGGRAPH Asia, 182:1–182:10.
7. Fisher, M., Savva, M., and Hanrahan, P. 2011. Characterizing structural relationships in scenes using graph kernels. ACM TOG 30, 4, 34.
8. Fisher, M., Ritchie, D., Savva, M., Funkhouser, T. A., and Hanrahan, P. 2012. Example-based synthesis of 3d object arrangements. ACM TOG 31, 6, 135.
9. Fisher, M., Savva, M., Li, Y., Hanrahan, P., and Nie\s sner, M. 2015. Activity-centric scene synthesis for functional 3d scene modeling. ACM TOG 34, 6, 179.
10. Funkhouser, T., Kazhdan, M., Shilane, P., Min, P., Kiefer, W., Tal, A., Rusinkiewicz, S., and Dobkin, D. 2004. Modeling by example. ACM TOG 23, 3, 652–663.
11. Gal, R., and Cohen-Or, D. 2006. Salient geometric features for partial shape matching and similarity. ACM TOG 25, 1, 130–150.
12. Hu, R., Zhu, C., van Kaick, O., Liu, L., Shamir, A., and Zhang, H. 2015. Interaction context (icon): Towards a geometric functionality descriptor. ACM TOG 34, 4, 83:1–83:12.
13. Hu, R., van Kaick, O., Wu, B., Huang, H., Shamir, A., and Zhang, H. 2016. Learning how objects function via co-analysis of interactions. ACM SIGGRAPH 35, 4 (July), 47:1–47:13.
14. Huang, S. S., Fu, H., Wei, L. Y., and Hu, S. M. 2016. Support substructures: Support-induced part-level structural representation. IEEE TVCG 22, 8 (Aug), 2024–2036.
15. Hunter, D. R. 2004. Mm algorithms for generalized bradleyterry models. Ann. Statist. 32, 1 (02), 384–406.
16. Jiang, Y., Koppula, H. S., and Saxena, A. 2013. Hallucinated humans as the hidden context for labeling 3d scenes. In IEEE CVPR.
17. Johnson, A. E., and Hebert, M. 1999. Using spin images for efficient object recognition in cluttered 3d scenes. IEEE PAMI 21, 5, 433–449.
18. Kalogerakis, E., Chaudhuri, S., Koller, D., and Koltun, V. 2012. A probabilistic model for component-based shape synthesis. ACM TOG 31, 4, 55.
19. Kazhdan, M., Funkhouser, T., and Rusinkiewicz, S. 2003. Rotation invariant spherical harmonic representation of 3 d shape descriptors. In SGP.
20. Kazhdan, M., Simari, P., McNutt, T., Wu, B., Jacques, R., Chuang, M., and Taylor, R. 2009. A shape relationship descriptor for radiation therapy planning. In MICCAI 2009. Springer, 100–108.
21. Kim, V. G., Chaudhuri, S., Guibas, L., and Funkhouser, T. 2014. Shape2Pose: Human-centric shape analysis. ACM TOG 33, 4.
22. Kostelec, P. J., 2008. SpharmonicKit.
23. Liu, T., Chaudhuri, S., Kim, V. G., Huang, Q.-X., Mitra, N. J., and Funkhouser, T. 2014. Creating consistent scene graphs using a probabilistic grammar. ACM TOG 33, 6.
24. Majerowicz, L., Shamir, A., Sheffer, A., and Hoos, H. H. 2014. Filling your shelves: Synthesizing diverse style-preserving artifact arrangements. IEEE TVCG 20, 11, 1507–1518.
25. Mitra, N. J., Wand, M., Zhang, H., Cohen-Or, D., and Bokeloh, M. 2013. Structure-aware shape processing. In EUROGRAPHICS State-of-the-art Report.
26. Nan, L., Xie, K., and Sharf, A. 2012. A search-classify approach for cluttered indoor scene understanding. ACM TOG 31, 6, 137.
27. Saupe, D., and Vranic, D. V. 2001. 3d model retrieval with spherical harmonics and moments. In 23rd DAGM-Symposium on Pattern Recognition, Springer-Verlag, 392–397.
28. Savva, M., Chang, A. X., Hanrahan, P., Fisher, M., and Niessner, M. 2014. Scenegrok: Inferring action maps in 3d environments. ACM TOG 33, 6.
29. Savva, M., Chang, A. X., Hanrahan, P., Fisher, M., and Niessner, M. 2016. Pigraphs: Learning interaction snapshots from observations. ACM SIGGRAPH 35, 4 (July), 139:1–139:12.
30. Shao, T., Xu, W., Zhou, K., Wang, J., Li, D., and Guo, B. 2012. An interactive approach to semantic modeling of indoor scenes with an rgbd camera. ACM TOG 31, 6, 136:1–136:11.
31. Shapira, L., Shamir, A., and Cohen-Or, D. 2008. Consistent mesh partitioning and skeletonisation using the shape diameter function. The Visual Computer 24, 4 (Apr.), 249–259.
32. Su, X., Chen, X., Fu, Q., and Fu, H. 2016. Cross-class 3d object synthesis guided by reference examples. Computers & Graphics 54, 145 — 153.
33. van Kaick, O., Xu, K., Zhang, H., Wang, Y., Sun, S., Shamir, A., and Cohen-Or, D. 2013. Co-hierarchical analysis of shape structures. ACM TOG 32, 4, 69:1–69:10.
34. van Kaick, O., Zhang, H., and Hamarneh, G. 2013. Bilateral maps for partial matching. In CGF, Wiley Online Library.
35. Wang, Y., Xu, K., Li, J., Zhang, H., Shamir, A., Liu, L., Cheng, Z.-Q., and Xiong, Y. 2011. Symmetry hierarchy of man-made objects. CGF 30, 2, 287–296. Cross Ref
36. Wolfson, H. J., and Rigoutsos, I. 1997. Geometric hashing: An overview. Computational Science & Engineering, IEEE 4, 4, 10–21.
37. Xu, K., Ma, R., Zhang, H., Zhu, C., Shamir, A., Cohen-Or, D., and Huang, H. 2014. Organizing heterogeneous scene collection through contextual focal points. ACM TOG 33, 4.
38. Yeh, Y.-T., Yang, L., Watson, M., Goodman, N. D., and Hanrahan, P. 2012. Synthesizing open worlds with constraints using locally annealed reversible jump mcmc. ACM TOG 31, 4, 56.
39. Yu, L.-F., Yeung, S. K., Tang, C.-K., Terzopoulos, D., Chan, T. F., and Osher, S. 2011. Make it home: automatic optimization of furniture arrangement. ACM TOG 30, 4, 86.
40. Zhao, X., Wang, H., and Komura, T. 2014. Indexing 3d scenes using the interaction bisector surface. ACM TOG 33, 5.
41. Zheng, B., Ishikawa, R., Oishi, T., Takamatsu, J., and Ikeuchi, K. 2009. A fast registration method using IP and its application to ultrasound image registration. IPSJ Trans. Comp. Vis. and App. 1, 209–219. Cross Ref
42. Zheng, Y., Cohen-Or, D., and Mitra, N. J. 2013. Smart variations: Functional substructures for part compatibility. CGF 32, 2pt2, 195–204.
43. Zheng, Y., Tai, C.-L., Zhang, E., and Xu, P. 2013. Pairwise harmonics for shape analysis. IEEE TVCG 19, 7, 1172–1184.
