“Dream Painter: Exploring creative possibilities of AI-aided speech-to-image synthesis in the interactive art context” by Sola and Guljajeva

  • ©Mar Canet Sola and Varvara Guljajeva

Conference:


Type:


Entry Number: 01

Title:

    Dream Painter: Exploring creative possibilities of AI-aided speech-to-image synthesis in the interactive art context

Presenter(s)/Author(s):


Abstract:


    This paper describes an interactive robotic art installation Dream Painter by the artistic duo Varvara & Mar that deploys artificial intelligence (AI), a KUKA industrial robot and interaction technology in order to offer the audience an artistic interpretation of their past dreams, which are then turned into a collective painting. The installation is composed of four larger parts: audience interaction design, AI-driven multicoloured drawing software, communication with an arm robot, and a kinetic part that is the automatic paper progression following each completed dream drawing. All these interconnected parts are orchestrated into an interactive and autonomous system in the form of an art installation that occupies two floors of a cultural centre. In the article, we document the technical and conceptual frameworks of the project, and the experience gained through the creation and exhibition of the interactive robotic art installation. In addition, the paper explores the creative potential of speech-to-AI-drawing transformation, which is a translation of different semiotic spaces performed by a robot as a method for audience interaction in the art exhibition context.

References:


    1. Paul Cohen. 2016. Harold Cohen and AARON. Ai Magazine 37, 4 (2016), 63–66.
    2. Kevin Frans, Lisa Soros, and Olaf Witkowski. 2021. Clipdraw: Exploring text-to-drawing synthesis through language-image encoders. arXiv preprint arXiv:2106.14843 (2021).
    3. Madeline Gannon. 2017. Human-centered Interfaces for autonomous fabrication machines.
    4. Benjamin Grosser. 2011. Interactive Robotic Painting Machine. Retrieved April 2, 2022 from https://bengrosser.com/projects/interactive-robotic-painting-machine/
    5. Varvara Guljajeva. 2018. From Interaction to Post-participation: the Disappearing Role of the Active Participant. Ph.D. Dissertation.
    6. Varvara Guljajeva. 2021. Synthetic Books. In 10th International Conference on Digital and Interactive Arts. 1–7.
    7. Aaron Hertzmann. 2019. Aesthetics of neural network art. arXiv preprint arXiv:1903.05696 (2019).
    8. Aaron Hertzmann. 2020. Visual indeterminacy in GAN art. Leonardo 53, 4 (2020), 424–428.
    9. Indrek Ibrus and Maarja Ojamaa. 2020. The Creativity of Digital (Audiovisual) Archives: A Dialogue Between Media Archaeology and Cultural Semiotics. Theory, Culture & Society 37, 3 (2020), 49–70.
    10. Tzu-Mao Li, Michal Lukáč, Michaël Gharbi, and Jonathan Ragan-Kelley. 2020. Differentiable vector graphics rasterization for editing and learning. ACM Transactions on Graphics (TOG) 39, 6 (2020), 1–15.
    11. Yuri M Lotman. 1990. Universe of the mind: A semiotic theory of culture. Indiana University Press. 143 pages.
    12. Daniel Michelis and Jörg Müller. 2011. The audience funnel: Observations of gesture based interaction with multiple large displays in a city center. Intl. Journal of Human-Computer Interaction 27, 6 (2011), 562–579.
    13. Simone Natale and Andrea Ballatore. 2020. Imagining the thinking machine: Technological myths and the rise of artificial intelligence. Convergence 26, 1 (2020), 3–18.
    14. Changhoon Oh, Jungwoo Song, Jinhan Choi, Seonghyeon Kim, Sungwoo Lee, and Bongwon Suh. 2018. I lead, you help but only with enough details: Understanding user experience of co-creation with artificial intelligence. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–13.
    15. Fred Plaut. 2000. FREUD, SIGMUND. The Interpretation of Dreams. A New Translation by Joyce Crick. Oxford & New York: Oxford University Press, 1999. Journal of Analytical Psychology 45, 4 (2000), 649–651.
    16. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. PMLR, 8748–8763.
    17. Eleanor Sandry. 2017. Creative collaborations with machines. Philosophy & Technology 30, 3 (2017), 305–319.
    18. Peter Schaldenbrand, Zhixuan Liu, and Jean Oh. 2021. StyleCLIPDraw: Coupling Content and Style in Text-to-Drawing Synthesis. arXiv preprint arXiv:2111.03133 (2021).
    19. Uwe Seifert and Jin Hyun Kim. 2008. Towards a conceptual framework and an empirical methodology in research on artistic human-computer and human-robot interaction. In Human Computer Interaction. Citeseer.
    20. Patrick Tresset and Oliver Deussen. 2014. Artistically skilled embodied agents. In AISB.
    21. Patrick Tresset and Frederic Fol Leymarie. 2012. Sketches by Paul the robot. In Proceedings of the eighth annual symposium on computational aesthetics in graphics, visualization, and imaging. 17–24.
    22. Josh Urban Davis, Fraser Anderson, Merten Stroetzel, Tovi Grossman, and George Fitzmaurice. 2021. Designing Co-Creative AI for Virtual Environments. In Creativity and Cognition. 1–11.
    23. Joseph Weizenbaum. 1976. Computer power and human reason: From judgment to calculation. (1976).

ACM Digital Library Publication:


Overview Page:


Art Paper/Presentation Type: