“LiveGS: Live Free-Viewpoint Video via High-Performance Gaussian Splatting for Mobile Devices” by Chen, Liang, Wang, Wang, Xie, et al. …
Conference:
Experience Type(s):
Title:
- LiveGS: Live Free-Viewpoint Video via High-Performance Gaussian Splatting for Mobile Devices
Organizer(s)/Presenter(s):
Interest Areas(s):
- New Technologies
Description:
In this paper, we present LiveGS, an innovative real-time free-viewpoint video (FVV) live-broadcasting system that addresses key challenges in live volumetric video streaming. Our system demonstrates three significant technical advancements: (1) a novel view synthesis framework capable of generating high-fidelity volumetric human 3D Gaussian representations with real-time inference, eliminating the need for per-scene optimization and specifically designed for live streaming applications; (2) an efficient compression scheme that maps temporal 3D Gaussian representations onto 2D video planes, enabling transmission through standard video codecs with minimal quality degradation; and (3) a region-based Gaussian modeling strategy applies coarse granularity to low-frequency regions and fine granularity for high-frequency regions, significantly reducing computational complexity on mobile devices by optimizing the number of 3D Gaussians required for accurate representation. The proposed system establishes a complete pipeline that enables real-time FVV playback on resource-constrained mobile devices while maintaining low transmission bandwidth requirements. Through extensive experimentation and system evaluation, we demonstrate that LiveGS represents the first practical solution for FVV live-broadcasting, featuring server-based temporal 3D Gaussian reconstruction, compression and transmission, while rendering can performed on both PC and mobile platforms. This breakthrough significantly advances the state-of-the-art in mobile FVV systems by achieving an unprecedented balance between visual fidelity, computational efficiency, and transmission overhead.
References:
[1] Vaswani Ashish, Shazeer Noam, Parmar Niki, Uszkoreit Jakob, Jones Llion, N. Gomez Aidan, Kaiser Lukasz, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems.
[2] Shaohui Jiao, Dehao Zhao, Yuzhong Chen, Shengdong Wang, Cong Xie, Bo Zhu, Ali Israr, and Li Zhang. 2024. Intractable Live Free-Viewpoint Video with Haptic Feedback. In ACM SIGGRAPH 2024 Immersive Pavilion. 1–2.
[3] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 2023. 3d gaussian splatting for real-time radiance field rendering.ACM Trans. Graph. 42, 4 (2023), 1–14.
[4] Lahav Lipson, Zachary Teed, and Jia Deng. 2021. RAFT-Stereo: Multilevel Recurrent Field Transforms for Stereo Matching. In International Conference on 3D Vision (3DV).
[5] MATTI SIEKKINEN and TEEMU KÄMÄRÄINEN. 2023. Neural Network Assisted Depth Map Packing for Compression Using Standard Hardware Video Codecs. In ACM Transactions on Multimedia Computing, Communications and Applications.
[6] Shunyuan Zheng, Boyao Zhou, Ruizhi Shao, Boning Liu, Shengping Zhang, Liqiang Nie, and Yebin Liu. 2024. GPS-Gaussian: Generalizable Pixel-wise 3D Gaussian Splatting for Real-time Human Novel View Synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
