“3D-Layers: Bringing Layer-based Color Editing to VR Painting”
Conference:
Type(s):
Title:
- 3D-Layers: Bringing Layer-based Color Editing to VR Painting
Presenter(s)/Author(s):
Abstract:
We introduce 3D-Layers, a new data structure and companion rendering algorithm to perform
layered painting in VR. We define 3D-Layers as groups of 3D strokes, and we distinguish the ones that represent 3D geometry from the ones that represent color modifications of the geometry.
References:
[1]
Adobe. 1990. Photoshop. https://www.adobe.com/products/photoshop.html.
[2]
Adobe. 2023a. Layer Masks. https://helpx.adobe.com/photoshop/using/masking-layers.html.
[3]
Adobe. 2023b. Techniques for nondestructive editing. https://helpx.adobe.com/photoshop/using/nondestructive-editing.html.
[4]
Maneesh Agrawala, Andrew C Beers, and Marc Levoy. 1995. 3D painting on scanned surfaces. In Proceedings of the 1995 symposium on Interactive 3D graphics. 145–ff.
[5]
Ya??z Aksoy, Tun? Ozan Ayd?n, Aljo?a Smoli?, and Marc Pollefeys. 2017. Unmixing-Based Soft Color Segmentation for Image Manipulation. ACM Transactions on Graphics 36, 2 (2017), 19:1–19:19.
[6]
Rahul Arora, Rubaiat Habib Kazi, Tovi Grossman, George Fitzmaurice, and Karan Singh. 2018. SymbiosisSketch: Combining 2D & 3D sketching for designing detailed 3D objects in situ. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–15.
[7]
Rahul Arora, Rubaiat Habib Kazi, Fraser Anderson, Tovi Grossman, Karan Singh, and George W Fitzmaurice. 2017. Experimental Evaluation of Sketching on Surfaces in VR. In CHI, Vol. 17. ACM, 5643–5654.
[8]
Rahul Arora and Karan Singh. 2021. Mid-air drawing of curves on 3D surfaces in virtual reality. ACM Transactions on Graphics (TOG) 40, 3 (2021), 1–17.
[9]
Seok-Hyung Bae, Ravin Balakrishnan, and Karan Singh. 2008. ILoveSketch: as-natural-as-possible sketching system for creating 3D curve models. In Proceedings of the 21st annual ACM symposium on User interface software and technology. 151–160.
[10]
Ilya Baran, Johannes Schmid, Thomas Siegrist, Markus Gross, and Robert W Sumner. 2011. Mixed-order compositing for 3D paintings. In Proceedings of the 2011 SIGGRAPH Asia Conference. 1–6.
[11]
Katie Bassett, Ilya Baran, Johannes Schmid, Markus Gross, and Robert W. Sumner. 2013. Authoring and animating painterly characters. ACM Transactions on Graphics 32, 5 (Sept. 2013), 1–12.
[12]
David Benson and Joel Davis. 2002. Octree Textures. ACM Trans. Graph. 21, 3 (jul 2002), 785–790.
[13]
Yifei Cheng, Yukang Yan, Xin Yi, Yuanchun Shi, and David Lindlbauer. 2021. Semanticadapt: Optimization-based adaptation of mixed reality layouts leveraging virtual-physical semantic connections. In The 34th Annual ACM Symposium on User Interface Software and Technology. 282–297.
[14]
Gianmarco Cherchi, Fabio Pellacini, Marco Attene, and Marco Livesu. 2022. Interactive and Robust Mesh Booleans. ACM Transactions on Graphics (TOG) 41, 6 (2022), 1–14.
[15]
Erin Cherry and Celine Latulipe. 2014. Quantifying the creativity support of digital tools through the creativity support index. ACM Transactions on Computer-Human Interaction (TOCHI) 21, 4 (2014), 1–25.
[16]
Franklin C Crow. 1977. Shadow algorithms for computer graphics. ACM Siggraph 11, 2 (1977), 242–248.
[17]
David DeBry, Jonathan Gibbs, Devorah DeLeon Petty, and Nate Robins. 2002. Painting and rendering textures on unparameterized models. In Proceedings of the 29th annual conference on Computer graphics and interactive techniques. 763–768.
[18]
Michael F Deering. 1995. HoloSketch: a virtual reality sketching/animation tool. ACM Transactions on Computer-Human Interaction (TOCHI) 2, 3 (1995), 220–238.
[19]
Julie Dorsey, Songhua Xu, Gabe Smedresman, Holly Rushmeier, and Leonard McMillan. 2007. The mental canvas: A tool for conceptual architectural design and analysis. In 15th Pacific Conference on Computer Graphics and Applications (PG’07). IEEE, 201–210.
[20]
Tobias Drey, Jan Gugenheimer, Julian Karlbauer, Maximilian Milo, and Enrico Rukzio. 2020. VRSketchIn: Exploring the Design Space of Pen and Tablet Interaction for 3D Sketching in Virtual Reality. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. ACM, 1–14.
[21]
Zheng-Jun Du, Liang-Fu Kang, Jianchao Tan, Yotam Gingold, and Kun Xu. 2023. Image vectorization and editing via linear gradient layer decomposition. ACM Transactions on Graphics 42, 4 (2023), 1–13.
[22]
Elmar Eisemann, Sylvain Paris, and Fredo Durand. 2009. A visibility algorithm for converting 3D meshes into editable 2D vector graphics. ACM Transactions on Graphics (Proc. SIGGRAPH) 28 (2009).
[23]
Elmar Eisemann, Holger Winnemoeller, John C. Hart, and David Salesin. 2008. Stylized vector art from 3d models with region support. Computer Graphics Forum (Proc. of EGSR) 27, 4 (June 2008).
[24]
Hesham Elsayed, Mayra Donaji Barrera Machuca, Christian Schaarschmidt, Karola Marky, Florian M?ller, Jan Riemann, Andrii Matviienko, Martin Schmitz, Martin Weigel, and Max M?hlh?user. 2020. VRsketchpen: unconstrained haptic assistance for sketching in virtual 3D environments. In Proceedings of the 26th ACM Symposium on Virtual Reality Software and Technology. 1–11.
[25]
Jo?o Marcelo Evangelista Belo, Mathias N. Lystb?k, Anna Maria Feit, Ken Pfeuffer, Peter K?n, Antti Oulasvirta, and Kaj Gr?nb?k. 2022. AUIT – the Adaptive User Interfaces Toolkit for Designing XR Applications (UIST ’22). Association for Computing Machinery.
[26]
Jean-Dominique Favreau, Florent Lafarge, and Adrien Bousseau. 2017. Photo2ClipArt: Image Abstraction and Vectorization Using Layered Linear Gradients. ACM Transactions on Graphics (Proc. SIGGRAPH Asia Conference) 36, 6 (November 2017).
[27]
Jack Goldfeather, Jeff P M Hultquist, and Henry Fuchs. 1986. Fast Constructive-Solid Geometry Display in the Pixel-Powers Graphics System. In Proc. Annual Conference on Computer Graphics and Interactive Techniques (ACM SIGGRAPH).
[28]
Gravity Sketch. 2017. Gravity Sketch. https://www.gravitysketch.com/.
[29]
Pat Hanrahan and Paul Haeberli. 1990. Direct WYSIWYG painting and texturing on 3D shapes. ACM SIGGRAPH computer graphics 24, 4 (1990), 215–223.
[30]
Sandra G Hart. 2006. NASA-task load index (NASA-TLX); 20 years later. In Proceedings of the human factors and ergonomics society annual meeting, Vol. 50. Sage publications Sage CA: Los Angeles, CA, 904–908.
[31]
Rorik Henrikson, Bruno Araujo, Fanny Chevalier, Karan Singh, and Ravin Balakrishnan. 2016a. Multi-Device Storyboards for Cinematic Narratives in VR. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. ACM, 787–796.
[32]
Rorik Henrikson, Bruno De Araujo, Fanny Chevalier, Karan Singh, and Ravin Balakrishnan. 2016b. Storeoboard: Sketching Stereoscopic Storyboards. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 4587–4598.
[33]
Laura M. Herman and Stefanie Hutka. 2019. Virtual Artistry: Virtual Reality Translations of Two-Dimensional Creativity. In Proceedings of the 2019 on Creativity and Cognition (C&C ’19). Association for Computing Machinery, 612–618.
[34]
Perry Hurt. 2013. Never Underestimate the Power of a Paint Tube. https://www.smithsonianmag.com/arts-culture/never-underestimate-the-power-of-a-paint-tube-36637764/. Smithsonian magazine (2013).
[35]
Icosa. 2020. Open Brush. https://openbrush.app/.
[36]
Savage Interactive. 2011. Procreate. https://procreate.com/.
[37]
Johann Habakuk Israel, Eva Wiese, Magdalena Mateescu, Christian Z?llner, and Rainer Stark. 2009. Investigating three-dimensional sketching for early conceptual design—Results from expert discussions and user studies. Computers & Graphics 33, 4 (2009), 462–473.
[38]
Bret Jackson and Daniel F Keefe. 2016. Lift-off: Using reference imagery and freehand sketching to create 3D models in VR. IEEE transactions on visualization and computer graphics 22, 4 (2016), 1442–1451.
[39]
Ying Jiang, Congyi Zhang, Hongbo Fu, Alberto Cannav?, Fabrizio Lamberti, Y K Henry Lau, and Wenping Wang. 2021. HandPainter – 3D Sketching in VR with Hand-based Physical Proxy. In ACM Conference on Human Factors in Computing Systems (CHI). ACM.
[40]
Robert D. Kalnins, Lee Markosian, Barbara J. Meier, Michael A. Kowalski, Joseph C. Lee, Philip L. Davidson, Matthew Webb, John F. Hughes, and Adam Finkelstein. 2002. WYSIWYG NPR: Drawing Strokes Directly on 3D Models. ACM Transactions on Graphics (Proc. SIGGRAPH) 21, 3 (July 2002), 755–762.
[41]
George Katanics and Tasso Lappas. 2003. Deep Canvas: Integrating 3D Painting and Painterly Rendering. In Theory and Practice of Non-Photorealistic Graphics: Algorithms, Methods, and Production Systems.
[42]
Yeojin Kim, Byungmoon Kim, and Young J Kim. 2018. Dynamic deep octree for high-resolution volumetric painting in virtual reality. In Computer Graphics Forum, Vol. 37. Wiley Online Library, 179–190.
[43]
Joon Hyub Lee, Hanbit Kim, and Seok-Hyung Bae. 2022. Rapid design of articulated objects. ACM Transactions on Graphics (TOG) 41, 4 (2022), 1–8. Publisher: ACM New York, NY, USA.
[44]
Jorge Lopez-Moreno, Popov Stefan, Adrien Bousseau, Maneesh Agrawala, and George Drettakis. 2013. Depicting stylized materials with vector shade trees. ACM Transactions on Graphics 32, 4 (2013).
[45]
Mayra Donaji Barrera Machuca, Wolfgang Stuerzlinger, and Paul Asente. 2019. The Effect of Spatial Ability on Immersive 3D Drawing. In Proceedings of the ACM Conference on Creativity & Cognition (C&C’19). ACM, 173–186.
[46]
Wendy E Mackay. 2023. DOIT: The Design of Interactive Things. Selected methods for quickly and effectively designing interactive systems from the user’s perspective. In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems. 1–3.
[47]
Priya Mistry. 2022. Journey. https://www.priyamistry.co.uk/portfolio/product/departures-journey/.
[48]
Brad A Myers, Ashley Lai, Tam Minh Le, YoungSeok Yoon, Andrew Faulring, and Joel Brandt. 2015. Selective undo support for painting applications. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. 4227–4236.
[49]
Michael Nebeling, Katy Lewis, Yu-Cheng Chang, Lihan Zhu, Michelle Chung, Piaoyang Wang, and Janet Nebeling. 2020. XRDirector: A Role-Based Collaborative Immersive Authoring System. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. ACM, 1–12.
[50]
NVRMIND. 2018. AnimVR. https://www.meta.com/en-gb/experiences/pcvr/1741124389277542/.
[51]
Lohit Petikam, Ken Anjyo, and Taehyun Rhee. 2021. Shading rig: Dynamic art-directable stylised shading for 3D characters. ACM Transactions on Graphics (TOG) 40, 5 (2021), 1–14.
[52]
Owen D. Pomery. 2021. Owen Pomery – AIRBNB. https://owenpomery.com/airbnb.
[53]
Thomas Porter and Tom Duff. 1984. Compositing digital images. SIGGRAPH 18, 3 (1984), 253–259.
[54]
Gonzalo Ramos, George Robertson, Mary Czerwinski, Desney Tan, Patrick Baudisch, Ken Hinckley, and Maneesh Agrawala. 2006. Tumble! Splat! Helping Users Access and Manipulate Occluded Content in 2D Drawings. In Proceedings of the Working Conference on Advanced Visual Interfaces (AVI ’06). Association for Computing Machinery, 428–435.
[55]
Christian Remy, Lindsay MacDonald Vermeulen, Jonas Frich, Michael Mose Biskjaer, and Peter Dalsgaard. 2020. Evaluating creativity support tools in HCI research. In Proceedings of the 2020 ACM designing interactive systems conference. 457–476.
[56]
Miguel A Renom, Baptiste Caramiaux, and Michel Beaudouin-Lafon. 2023. Interaction Knowledge: Understanding the ‘Mechanics’ of Digital Tools. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–14.
[57]
Christian Richardt, Jorge Lopez-Moreno, Adrien Bousseau, Maneesh Agrawala, and George Drettakis. 2014. Vectorising Bitmaps into Semi-Transparent Gradient Layers. Computer Graphics Forum (Proc. EGSR) 33, 4 (July 2014), 11–19.
[58]
Scott Robertson and Thomas Bertling. 2014. How to Render: the fundamentals of light, shadow and reflectivity.
[59]
Enrique Rosales, Chrystiano Ara?jo, Jafet Rodriguez, Nicholas Vining, Dongwook Yoon, and Alla Sheffer. 2021. AdaptiBrush: Adaptive General and Predictable VR Ribbon Brush. ACM Transaction on Graphics (Proc. SIGGRAPH Asia) 40, 1 (2021).
[60]
Enrique Rosales, Jafet Rodriguez, and Alla Sheffer. 2019. SurfaceBrush: From Virtual Reality Drawings to Manifold Surfaces. ACM Transaction on Graphics 38, 4, Article 96 (2019), 15 pages.
[61]
Steven Schkolne, Michael Pruett, and Peter Schr?der. 2001. Surface drawing: creating organic 3D shapes with the hand and tangible tools. In Proceedings of the SIGCHI conference on Human factors in computing systems. 261–268.
[62]
Johannes Schmid, Martin Sebastian Senn, Markus Gross, and Robert W Sumner. 2011. OverCoat: an implicit canvas for 3D painting. In ACM SIGGRAPH 2011 papers. 1–10.
[63]
Evan Shimizu, Matt Fisher, Sylvain Paris, and Kayvon Fatahalian. 2019. Finding Layers Using Hover Visualizations. In Proceedings of the 45th Graphics Interface Conference on Proceedings of Graphics Interface 2019. 1–9.
[64]
Sketchsoft. 2022. Feather: 3D Sketchbook. https://feather.art
[65]
Smoothstep. 2021. Quill. https://quill.art/.
[66]
Jianchao Tan, Marek Dvoro???k, Daniel S?kora, and Yotam Gingold. 2015. Decomposing Time-Lapse Paintings into Layers. ACM Transactions on Graphics (TOG) 34, 4, Article 61 (July 2015), 10 pages.
[67]
Jianchao Tan, Jyh-Ming Lien, and Yotam Gingold. 2016. Decomposing Images into Layers via RGB-space Geometry. ACM Transactions on Graphics (TOG) 36, 1 (Nov. 2016).
[68]
Unity. 2023. Unity Engine. https://unity.com/products/unity-engine.
[69]
W3C. 2023. Compositing and Blending. https://www.w3.org/TR/compositing/.
[70]
Xue Yu, Stephen DiVerdi, Akshay Sharma, and Yotam Gingold. 2021. ScaffoldSketch: Accurate Industrial Design Drawing in VR. In Proceedings of ACM Symposium on User Interface Software and Technology (UIST).
