“Neural Performance Toolset: AI-Powered Human Performance Synthesis” by Plaete, Olivieri-Dancey, Frigo, Anton, Correa, et al. …
Conference:
Type(s):
Title:
- Neural Performance Toolset: AI-Powered Human Performance Synthesis
Session/Category Title:
- Generative Intelligence
Presenter(s)/Author(s):
- Jo Plaete
- Matteo Olivieri-Dancey
- Oriel Frigo
- Mihai Anton
- Sebastian Correa
- Simon Deckers
- Thomas Salama
- Terrence Bannon
- Tomas Koutsky
Moderator(s):
Abstract:
The Metaphysic Neural Performance Toolset introduces a powerful framework for photorealistic, AI-driven human performance synthesis, designed for both cinematic and real-time productions. Built on custom autoencoder architectures, identity-specific training, and refined latent-space manipulation, it balances high-fidelity realism with creative flexibility. Its data-driven methodology—encompassing iterative hyperparameter tuning and ethically sourced datasets—ensures robust results across diverse scenarios, from feature films to live broadcasts. A key strength of the system lies in its fine-grained, post-capture editing capabilities. Through precise latent-space controls, directors and artists can adjust expressions, retime facial motions, or blend distinctive features across multiple takes, all while preserving core identity. Real-time previews with sub-100ms latency significantly accelerate on-set decision-making, empowering teams to assess changes instantly and make collaborative refinements. This immediate feedback loop fosters creative spontaneity, allowing actors and directors to iteratively shape performances with minimal downtime. Although primarily focused on visual synthesis, the toolset also includes a streamlined voice module for voice-to-voice transformations, enabling cohesive character performances that remain consistent across both face and voice. This unified approach provides a complete, adaptable pipeline for modern entertainment demands. Deployed in high-profile productions such as HERE, Alien: Romulus, and Mad Max: Furiosa, the technology has also been utilized in major live events featuring artists like Drake and Eminem. These diverse use cases demonstrate the toolset’s adaptability and reliability, whether integrated into a film post pipeline or employed in real-time concert environments.
