“Real-time compression and streaming of 4D performances” – ACM SIGGRAPH HISTORY ARCHIVES

“Real-time compression and streaming of 4D performances”

  • 2018 SA Technical Papers_Tang_Real-time compression and streaming of 4D performances

Conference:


Type(s):


Title:

    Real-time compression and streaming of 4D performances

Session/Category Title:   Capturing 4D performances

Presenter(s)/Author(s):


Moderator(s):


Abstract:


    We introduce a realtime compression architecture for 4D performance capture that is two orders of magnitude faster than current state-of-the-art techniques, yet achieves comparable visual quality and bitrate. We note how much of the algorithmic complexity in traditional 4D compression arises from the necessity to encode geometry using an explicit model (i.e. a triangle mesh). In contrast, we propose an encoder that leverages an implicit representation (namely a Signed Distance Function) to represent the observed geometry, as well as its changes through time. We demonstrate how SDFs, when defined over a small local region (i.e. a block), admit a low-dimensional embedding due to the innate geometric redundancies in their representation. We then propose an optimization that takes a Truncated SDF (i.e. a TSDF), such as those found in most rigid/non-rigid reconstruction pipelines, and efficiently projects each TSDF block onto the SDF latent space. This results in a collection of low entropy tuples that can be effectively quantized and symbolically encoded. On the decoder side, to avoid the typical artifacts of block-based coding, we also propose a variational optimization that compensates for quantization residuals in order to penalize unsightly discontinuities in the decompressed signal. This optimization is expressed in the SDF latent embedding, and hence can also be performed efficiently. We demonstrate our compression/decompression architecture by realizing, to the best of our knowledge, the first system for streaming a real-time captured 4D performance on consumer-level networks.

References:


    1. Sofien Bouaziz, Andrea Tagliasacchi, and Mark Pauly. 2013. Sparse Iterative Closest Point. Computer Graphics Forum (Symposium on Geometry Processing) (2013). Google ScholarDigital Library
    2. Stéphane Calderon and Tamy Boubekeur. 2017. Bounding proxies for shape approximation. ACM Transactions on Graphics (TOG) 36, 4 (2017), 57. Google ScholarDigital Library
    3. Daniel-Ricao Canelhas, Erik Schaffernicht, Todor Stoyanov, Achim J Lilienthal, and Andrew J Davison. 2017. Compressed Voxel-Based Mapping Using Unsupervised Learning. Robotics (2017).Google Scholar
    4. Paolo Cignoni, Claudio Rocchini, and Roberto Scopigno. 1998. Metro: Measuring error on simplified surfaces. (1998).Google Scholar
    5. David Cohen-Steiner, Pierre Alliez, and Mathieu Desbrun. 2004. Variational shape approximation. ACM Trans. on Graphics (Proc. of SIGGRAPH) (2004). Google ScholarDigital Library
    6. Alvaro Collet, Ming Chuang, Pat Sweeney, Don Gillett, Dennis Evseev, David Calabrese, Hugues Hoppe, Adam Kirk, and Steve Sullivan. 2015. High-quality streamable free-viewpoint video. ACM Trans. on Graphics (TOG) (2015). Google ScholarDigital Library
    7. Brian Curless and Marc Levoy. 1996. A volumetric method for building complex models from range images. In SIGGRAPH. Google ScholarDigital Library
    8. B. R. de Araujo, Daniel S. Lopes, Pauline Jepp, Joaquim A. Jorge, and Brian Wyvill. 2015. A Survey on Implicit Surface Polygonization. Comput. Surveys (2015). Google ScholarDigital Library
    9. Mingsong Dou, Philip Davidson, Sean Ryan Fanello, Sameh Khamis, Adarsh Kowdle, Christoph Rhemann, Vladimir Tankovich, and Shahram Izadi. 2017. Motion2fusion: real-time volumetric performance capture. ACM Trans. on Graphics (Proc. of SIGGRAPH Asia) (2017). Google ScholarDigital Library
    10. Mingsong Dou, Sameh Khamis, Yury Degtyarev, Philip Davidson, Sean Ryan Fanello, Adarsh Kowdle, Sergio Orts Escolano, Christoph Rhemann, David Kim, Jonathan Taylor, et al. 2016. Fusion4d: Real-time performance capture of challenging scenes. ACM Transactions on Graphics (TOG) (2016). Google ScholarDigital Library
    11. Frank Galligan, Michael Hemmer, Ondrej Stava, Fan Zhang, and Jamieson Brettle. 2018. Google/Draco: a library for compressing and decompressing 3D geometric meshes and point clouds. https://github.com/google/draco. (2018).Google Scholar
    12. Michael Garland and Paul S Heckbert. 1997. Surface simplification using quadric error metrics. In Proc. of ACM SIGGRAPH. Google ScholarDigital Library
    13. Allan Grønlund, Kasper Green Larsen, Alexander Mathiasen, Jesper Sindahl Nielsen, Stefan Schneider, and Mingzhou Song. 2017. Fast exact k-means, k-medians and bregman divergence clustering in 1d. arXiv preprint arXiv:1701.07204 (2017).Google Scholar
    14. Hugues Hoppe. 1996. Progressive Meshes. In Proc. of ACM SIGGRAPH. Google ScholarDigital Library
    15. Matthias Innmann, Michael Zollhöfer, Matthias Nießner, Christian Theobalt, and Marc Stamminger. 2016. VolumeDeform: Real-time volumetric non-rigid reconstruction. In Proc. of the European Conf. on Comp. Vision. 362–379.Google ScholarCross Ref
    16. Palle ET Jorgensen and Myung-Sin Song. 2007. Entropy encoding, Hilbert space, and Karhunen-Loeve transforms. J. Math. Phys. 48, 10 (2007), 103503.Google ScholarCross Ref
    17. Zachi Karni and Craig Gotsman. 2000. Spectral compression of mesh geometry. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques. ACM Press/Addison-Wesley Publishing Co. Google ScholarDigital Library
    18. Zachi Karni and Craig Gotsman. 2004. Compression of soft-body animation sequences. Computers & Graphics (2004).Google Scholar
    19. Michael Kazhdan and Hugues Hoppe. 2013. Screened poisson surface reconstruction. ACM Transactions on Graphics (ToG) (2013). Google ScholarDigital Library
    20. Adarsh Kowdle, Christoph Rhemann, Sean Fanello, Andrea Tagliasacchi, Jonathan Taylor, Philip Davidson, Mingsong Dou, Kaiwen Guo, Cem Keskin, Sameh Khamis, David Kim, Danhang Tang, Vladimir Tankovich, Julien Valentin, and Shahram Izadi. 2018. The Need 4 Speed in Real-Time Dense Visual Tracking. (2018).Google Scholar
    21. Bruno Lévy and Hao (Richard) Zhang. 2010. Spectral Mesh Processing. In ACM SIGGRAPH Courses. Google ScholarDigital Library
    22. William E Lorensen and Harvey E Cline. 1987. Marching cubes: A high resolution 3D surface construction algorithm. In Computer Graphics (Proc. SIGGRAPH), Vol. 21. 163–169. Google ScholarDigital Library
    23. Adrien Maglo, Guillaume Lavoué, Florent Dupont, and Céline Hudelot. 2015. 3d mesh compression: Survey, comparisons, and emerging trends. Comput. Surveys (2015). Google ScholarDigital Library
    24. G Nigel N Martin. 1979. Range encoding: an algorithm for removing redundancy from a digitised message. In Proc. IERE Video & Data Recording Conf., 1979.Google Scholar
    25. MPEG4/AFX. 2008. ISO/IEC 14496-16: MPEG-4 Part 16, Animation Framework eXtension (AFX). Technical Report. The Moving Picture Experts Group. https://mpeg.chiariglione.org/standards/mpeg-4/animation-framework-extension-afxGoogle Scholar
    26. Richard A Newcombe, Dieter Fox, and Steven M Seitz. 2015. Dynamicfusion: Reconstruction and tracking of non-rigid scenes in real-time. In Proc. of Comp. Vision and Pattern Recognition (CVPR).Google ScholarCross Ref
    27. Sergio Orts-Escolano, Christoph Rhemann, Sean Fanello, Wayne Chang, Adarsh Kowdle, Yury Degtyarev, David Kim, Philip L Davidson, Sameh Khamis, Mingsong Dou, et al. 2016. Holoportation: Virtual 3d teleportation in real-time. In Proc. of the Symposium on User Interface Software and Technology. Google ScholarDigital Library
    28. Jingliang Peng, Chang-Su Kim, and C-C Jay Kuo. 2005. Technologies for 3D mesh compression: A survey. Journal of Visual Communication and Image Representation (2005). Google ScholarDigital Library
    29. Fabián Prada, Misha Kazhdan, Ming Chuang, Alvaro Collet, and Hugues Hoppe. 2017. Spatiotemporal atlas parameterization for evolving meshes. ACM Trans. on Graphics (TOG) (2017). Google ScholarDigital Library
    30. Iain E Richardson. 2011. The H. 264 advanced video compression standard. John Wiley & Sons. Google ScholarDigital Library
    31. Oren Rippel and Lubomir Bourdev. 2017. Real-time adaptive image compression. arXiv preprint arXiv:1705.05823 (2017).Google Scholar
    32. Jarek Rossignac. 1999. Edgebreaker: Connectivity compression for triangle meshes. IEEE Transactions on Visualization and Computer Graphics (1999). Google ScholarDigital Library
    33. Michael Ruhnke, Liefeng Bo, Dieter Fox, and Wolfram Burgard. 2013. Compact RGBD Surface Models Based on Sparse Coding. (2013).Google Scholar
    34. Peter Schelkens, Adrian Munteanu, Joeri Barbarien, Mihnea Galca, Xavier Giro-Nieto, and Jan Cornelis. 2003. Wavelet coding of volumetric medical datasets. IEEE Transactions on medical Imaging (2003).Google ScholarCross Ref
    35. Peter Schelkens, Adrian Munteanu, Alexis Tzannes, and Christopher M. Brislawn. 2006. JPEG2000 Part 10 – Volumetric data encoding. (2006).Google Scholar
    36. Olga Sorkine, Daniel Cohen-Or, and Sivan Toledo. 2003. High-Pass Quantization for Mesh Encoding.. In Proc. of the Symposium on Geometry Processing. Google ScholarDigital Library
    37. Robert W Sumner, Johannes Schmid, and Mark Pauly. 2007. Embedded manipulation for shape manipulation. ACM Trans. on Graphics (TOG) (2007). Google ScholarDigital Library
    38. Jean-Marc Thiery, Emilie Guy, and Tamy Boubekeur. 2013. Sphere-Meshes: Shape Approximation using Spherical Quadric Error Metrics. ACM Trans. on Graphics (Proc. of SIGGRAPH Asia) (2013). Google ScholarDigital Library
    39. Jean-Marc Thiery, Émilie Guy, Tamy Boubekeur, and Elmar Eisemann. 2016. Animated Mesh Approximation With Sphere-Meshes. ACM Trans. on Graphics (TOG) (2016). Google ScholarDigital Library
    40. Anastasia Tkach, Mark Pauly, and Andrea Tagliasacchi. 2016. Sphere-Meshes for RealTime Hand Modeling and Tracking. ACM Transaction on Graphics (Proc. SIGGRAPH Asia) (2016). Google ScholarDigital Library
    41. Sébastien Valette and Rémy Prost. 2004. Wavelet-based progressive compression scheme for triangle meshes: Wavemesh. IEEE Transactions on Visualization and Computer Graphics 10, 2 (2004), 123–129. Google ScholarDigital Library
    42. Libor Vasa and Vaclav Skala. 2011. A perception correlated comparison method for dynamic meshes. IEEE transactions on visualization and computer graphics (2011). Google ScholarDigital Library

ACM Digital Library Publication:


Overview Page:


Submit a story:

If you would like to submit a story about this presentation, please contact us: historyarchives@siggraph.org