“Beyond Digital Twins: 3D Gaussian Splatting, Game Engines and Crossmedia Cultural Heritage Representations” by Mirocha
Conference:
Type(s):
Title:
- Beyond Digital Twins: 3D Gaussian Splatting, Game Engines and Crossmedia Cultural Heritage Representations
Session/Category Title:
- Generative Intelligence
Presenter(s)/Author(s):
Moderator(s):
Abstract:
This talk contextualizes an integration of 3D Gaussian Splatting (3DGS) with game engines for cultural heritage visualization, using a historic Hakka mansion in Hong Kong as a case study. As an emerging technique for real-time rendering of photorealistic 3D reconstructions, 3DGS offers high visual fidelity but presents technical challenges, including pipeline compatibility issues, limited lighting interactions, geometry inconsistencies, and artifact generation. The pilot project investigates how these constraints can be leveraged creatively, positioning 3DGS not merely as a tool for digital twins but as a medium for crossmedia storytelling and experimental heritage visualization. The project consists of three stages: data capture, processing, and creative integration. Multi-device scanning—including DSLR photography, smartphone video, and SLAM-based LiDAR—was used to construct a comprehensive dataset. Processing was conducted through various 3DGS tools, refining the scans while embracing imperfections as part of the medium’s aesthetic. Finally, the processed data was integrated into real-time game engines (Unity and Unreal) to test visualization techniques, interactive exploration, and cinematic representation. Findings reveal that while 3DGS lacks the geometric precision of traditional 3D modeling, its ability to preserve detailed lighting and texture information enables a unique mode of spatial representation. By reframing technical artifacts as aesthetic features, we explore how emerging imaging workflows can become sites of speculative storytelling and aesthetics-driven preservation. The project demonstrates how hybrid workflows combining 3DGS with SLAM data can enhance spatial accuracy while maintaining the visual richness of neural rendering. Three experimental outputs—a 360° LED volume display animation, an interactive real-time experience, and a cinematic animation—showcase the potential of 3DGS for immersive heritage documentation. By exploring the intersection of 3DGS, game engines, and crossmedia, we contribute to emerging digital heritage methodologies, offering new ways to document, interpret, and experience historical as well as other types of physical sites and environments through real-time 3D technologies.
References:
[1] Aab. 2024. List of the 1,444 Historic Buildings with Assessment Results, The Antiquities Advisory Board, accessed November 15, 2024. https://www.aab.gov.hk/filemanager/aab/en/content_29/AAB- SM-chi.pdf
[2] Amo. 2024. Definition of Grading, The Antiquities Advisory Board, ac- cessed November 15, 2024. https://www.amo.gov.hk/en/historic-buildings/historic-buildings-hk/assessment/index.html
[3] Duygu Arican, Ferat Furkan Göksu, Nursu Tunalioglu, Taylan Ocalan. 2023. Research on 3D reconstruction of small size objects using struc- ture from motion photogrammetry via smartphone images. Journal of Geodesy and Geoinformation, 10, 112-123.
[4] Teun Dubbelman. 2011. Designing stories: practices of narrative in 3D computer games. In Proceedings of the 2011 ACM SIGGRAPH Symposium on Video Games (Sandbox ’11). Association for Computing Machinery, New York, NY, USA, 37–41.
[5] LumaAI Fab Unreal Plugin. 2024. https://www.fab.com/listings/b52460e0-3ace-465e-a378- 495a5531e318
[6] Fac. 2024. Hands-on Heritage: Shared Conservation of a Historic Hakka Mansion, accessed November 15, 2024. https://fac.arch.hku.hk/aclab/research/PunUk/ [12] [7] Jawset Unreal Plugin. 2024.
[8] Jawset Postshot (Beta). 2024.
[9] Antreas Kantaros, Theodore Ganetsos, Florian Ion Tiberiu. 2023. Three-Dimensional Printing and 3D Scanning: Emerging Technologies Exhibiting High Potential in the Field of Cultural Heritage. In Applied Sciences, 13(8), 4777.
[10] Jacek Kęsik, Kamil Żyła, Jerzy Montusiewicz, Marek Miłosz, Calin Neamtu, Marta Juszczyk. 2023. A Methodical Approach to 3D Scanning of Heritage Objects Being under Continuous Display. In Applied Sciences, 13(1), 441.
[11] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, George Drettakis, G. 2023. 3D Gaussian Splatting for Real-Time Radiance Field Rendering, In ACM Transactions on Graphics, 42(4), July. https://repo-sam.in-ria.fr/fungraph/3d-gaussian-splatting/
[12] LumaLabs. 2024. Luma AI Interactive Scenes (iOS/Web)
[13] Lu Lou, Yitian Li, Qi Zhang, Hanbing Wei. 2023. SLAM and 3D Semantic Reconstruction Based on the Fusion of Lidar and Monocular Vision. Sensors, 23(3), 1502.
[14] Lukasz Mirocha. 2021. Virtual Cameras and General-Purpose Game Engines: On Hybrid and Programmable Aesthetics of 3D Real-Time Environments.Ph.D. Dissertation. City University of Hong Kong.
[15] Niantic. 2024. Scaniverse.
[16] ArasP. 2024. UnityGaussianSplatting. https://github.com/aras-p/UnityGaussianSplatting
[17] PlayCanvas. 2024. SuperSplat. https://playcanvas.com/supersplat/editor/
[18] Xgrids LCC Unreal SDK. 2024. https://developer.xgrids.com/#/
[19] Xgrids LixelKity K1. 2024. https://xgrids.com/lixelk1
[20] Xgrids LCC Unity SDK. 2024. https://developer.xgrids.com/#/
[21] Xverse XV3DGS-UEPlugin. 2024. https://github.com/xverse-engine/XV3DGS-UEPlugin
