“SentimentVoice: Integrating Emotion AI and VR in Performing Arts” by Bryant, LaBella, Ryu, Cano, Popiela, et al. …
Conference:
Experience Type(s):
Title:
- SentimentVoice: Integrating Emotion AI and VR in Performing Arts
Presenter(s):
Description:
SentimentVoice is a live VR performance art project that repurposes emotion-tracking AI to amplify immigrant narratives. While emotion AI is often used for surveillance or commercial gain, SentimentVoice subverts this technology into an empathetic tool, actively listening to immigrant stories instead of exploiting them. The project integrates human voice and facial expression as core materials, analyzed through emotion AI utilizing face tracking, the ChatGPT API, voice analysis, and speech-to-text conversion. While technologies like lie detectors and facial recognition have been used to monitor and target marginalized communities, SentimentVoice reclaims emotion data as a medium for fostering empathy, deep connections, and mutual understanding. Rooted in oral history, the project collects real immigrant stories from the Richmond community, structured around five key locational memories: airports, schools, immigration offices, hospitals, and workplaces. These spaces represent critical moments of transition and adaptation in the immigrant experience. The collected stories are woven into a performance script, enacted by two actors—one wearing a VR headset and the other without—bridging virtual and actual spaces, past memories and present realities. As the actors navigate five VR locations, emotion AI responds in real time, generating visual (particles, textures, light changes) and sound elements that reflect their speech and emotional expression. This interactive experience exposes the often-unnoticed ways our facial expressions and voices are tracked, transforming AI-driven surveillance into a medium for storytelling and emotional resonance. At SIGGRAPH, SentimentVoice invites the public to share their emotional stories while exploring the VR locations through a headset. The project encourages audience participation, allowing them to engage with emotion AI in a way that is transparent and interactive, challenging conventional uses of AI-driven emotion tracking. Additionally, a video documentation of the live performance will be screened, providing insight into the immersive nature of the project and its impact on storytelling through technology.
