“Perception-aware modeling and fabrication of digital drawing tools” by Piovarči, Levin, Kaufman and Didyk

  • ©Michal Piovarči, David I. W. Levin, Danny M. Kaufman, and Piotr Didyk



Entry Number: 123


    Perception-aware modeling and fabrication of digital drawing tools

Session/Category Title:   Perception & Haptics




    Digital drawing is becoming a favorite technique for many artists. It allows for quick swaps between different materials, reverting changes, and applying selective modifications to finished artwork. These features enable artists to be more efficient and creative. A significant disadvantage of digital drawing is poor haptic feedback. Artists are usually limited to one surface and a few different stylus nibs, and while they try to find a combination that suits their needs, this is typically challenging. In this work, we address this problem and propose a method for designing, evaluating, and optimizing different stylus designs. We begin with collecting a representative set of traditional drawing tools. We measure their physical properties and conduct a user experiment to build a perceptual space that encodes perceptually-relevant attributes of drawing materials. The space is optimized to both explain our experimental data and correlate it with measurable physical properties. To embed new drawing tool designs into the space without conducting additional experiments and measurements, we propose a new, data-driven simulation technique for characterizing stylus-surface interaction. We finally leverage the perceptual space, our simulation, and recent advancements in multi-material 3D printing to demonstrate the application of our system in the design of new digital drawing tools that mimic traditional drawing materials.


    1. Michelle Annett, Fraser Anderson, Walter F Bischof, and Anoop Gupta. 2014. The pen is mightier: understanding stylus behaviour while inking on tablets. In Proceedings of Graphics Interface 2014. Canadian Information Processing Society, 193–200. Google ScholarDigital Library
    2. Atakan Arasan, Cagatay Basdogan, and Tevfik Metin Sezgin. 2013. Haptic stylus with inertial and vibro-tactile feedback. In 2013 World Haptics Conference (WHC). 425–430.Google ScholarCross Ref
    3. John F. Archard. 1953. Contact and Rubbing of Flat Surfaces. Journal of Applied Physics 24, 8 (1953), 981–988.Google ScholarCross Ref
    4. Brian Armstrong-Hélouvry. 1991. Control of Machines with Friction. Kluwer Academic Publishers, Boston, Massachusetts.Google Scholar
    5. Mordecai Avriel. 2003. Nonlinear Programming: Analysis and Methods. Dover Publications.Google Scholar
    6. Olivier Bau, Ivan Poupyrev, Ali Israr, and Chris Harrison. 2010. TeslaTouch: Electro-vibration for Touch Surfaces. In Proceedings of the 23Nd Annual ACM Symposium on User Interface Software and Technology (UIST ’10). ACM, New York, NY, USA, 283–292. Google ScholarDigital Library
    7. Sliman Bensmaïa, Mark Hollins, and Jeffrey Yau. 2005. Vibrotactile intensity and frequency information in the Pacinian system: A psychophysical model. Attention, Perception, & Psychophysics 67, 5 (2005), 828–841.Google ScholarCross Ref
    8. Wouter M Bergmann Tiest and Astrid ML Kappers. 2006. Analysis of haptic perception of materials by multidimensional scaling and physical measurements of roughness and compressibility. Acta psychologica 121, 1 (2006), 1–20.Google Scholar
    9. Florence Bertails-Descoubes, Florent Cadoux, Gilles Daviet, and Vincent Acary. 2011. A nonsmooth Newton solver for capturing exact Coulomb friction in fiber assemblies. ACM Transactions on Graphics 30, 1 (Feb. 2011), 6:1–6:14. Google ScholarDigital Library
    10. Brian Blood. 2009. Physics of musical instruments. (2009).Google Scholar
    11. Carlos Canudas de Wit, Henrik Olsson, Karl J. Astrom, and Pablo Lischinsky. 1995. A new model for control of systems with friction. IEEE Trans. Automat. Control 40, 3 (Mar 1995), 419–425.Google Scholar
    12. Desai Chen, David I. W. Levin, Wojciech Matusik, and Danny M. Kaufman. 2017. Dynamics-aware Numerical Coarsening for Fabrication Design. ACM Trans. Graph. 36, 4, Article 84 (July 2017), 15 pages. Google ScholarDigital Library
    13. Youngjun Cho, Andrea Bianchi, Nicolai Marquardt, and Nadia Bianchi-Berthouze. 2016. RealPen: Providing Realism in Handwriting Tasks on Touch Surfaces Using Auditory-Tactile Feedback. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST ’16). ACM, New York, NY, USA, 195–205. Google ScholarDigital Library
    14. Seungmoon Choi and Hong Z. Tan. 2005. Toward Realistic Haptic Rendering of Surface Textures. In ACM SIGGRAPH 2005 Courses (SIGGRAPH ’05). ACM, New York, NY, USA, Article 125. Google ScholarDigital Library
    15. Vasilios G Chouvardas, Amalia N Miliou, and Miltiadis K Hatalis. 2008. Tactile displays: Overview and recent advances. Displays 29, 3 (2008), 185–194.Google ScholarCross Ref
    16. Vasilios G Chouvardas, Amalia N Miliou, Miltiadis K Hatalis, et al. 2005. Tactile displays: a short overview and recent developments. In Proceedings of the 5th International Conference on Technology and Automation (ICTA’05). 246–251.Google Scholar
    17. Theresa Cooke, Sebastian Kannengiesser, Christian Wallraven, and Heinrich H Bülthoff.Google Scholar
    18. 2006. Object feature validation using visual and haptic similarity ratings. ACM Transactions on Applied Perception (TAP) 3, 3 (2006), 239–261. Google ScholarDigital Library
    19. Theresa Cooke, Christian Wallraven, and Heinrich H Bülthoff. 2010. Multidimensional scaling analysis of haptic exploratory procedures. ACM Transactions on Applied Perception (TAP) 7, 1 (2010), 7. Google ScholarDigital Library
    20. Philip R. Dahl. 1976. Solid Friction Damping of Mechanical Vibrations. AIAA Journal 14, 12 (1976), 1675–1682.Google ScholarCross Ref
    21. Jérémy Danna and Jean-Luc Velay. 2015. Basic and supplementary sensory feedback in handwriting. Frontiers in Psychology 6 (2015), 169.Google ScholarCross Ref
    22. Nicholas Fellion, Thomas Pietrzak, and Audrey Girouard. 2017. FlexStylus: leveraging flexion input for pen interaction. In UIST 2017 (UIST 2017). Québec, Canada, 2482 — 2489. Google ScholarDigital Library
    23. Claire Watson Garcia. 2003. Drawing for the Absolute and Utter Beginner. Watson-Guptill.Google Scholar
    24. Ioannis Gkioulekas, Bei Xiao, Shuang Zhao, Edward H Adelson, Todd Zickler, and Kavita Bala. 2013. Understanding the role of phase function in translucent appearance. ACM Transactions on Graphics (TOG) 32, 5 (2013), 147. Google ScholarDigital Library
    25. Avraham Harnoy, Bernard Friedland, and Simon Cohn. 2008. Modeling and measuring friction effects. IEEE Control Systems 28, 6 (Dec 2008), 82–91.Google Scholar
    26. Vincent Hayward and M Cruz-Hernandez. 2000. Tactile display device using distributed lateral skin stretch. In Proceedings of the haptic interfaces for virtual environment and teleoperator systems symposium, Vol. 69. ASME, 1309–1314.Google ScholarCross Ref
    27. Richard Helps and Clarissa Helps. 2016. Measuring Stylus and Tablet Performance for Usability in Sketching. In Proceedings of the 5th Annual Conference on Research in Information Technology. ACM, 19–24. Google ScholarDigital Library
    28. Marlis Hochbruck and Alexander Ostermann. 2010. Exponential integrators. Acta Numerica 19 (2010), 209–286.Google ScholarCross Ref
    29. Mark Hollins, Sliman Bensmaïa, Kristie Karlof, and Forrest Young. 2000. Individual differences in perceptual space for tactile textures: Evidence from multidimensional scaling. Perception & Psychophysics 62, 8 (2000), 1534–1544.Google ScholarCross Ref
    30. Mark Hollins, Florian Lorenz, and Daniel Harper. 2006. Somatosensory Coding of Roughness: The Effect of Texture Adaptation in Direct and Indirect Touch. Journal of Neuroscience 26, 20 (2006), 5582–5588.Google ScholarCross Ref
    31. Ali Israr, Seungmoon Choi, and Hong Z. Tan. 2006a. Detection Threshold and Mechanical Impedance of the Hand in a Pen-Hold Posture. In 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems. 472–477.Google Scholar
    32. Ali Israr, Hong Z. Tan, and Charlotte M. Reed. 2006b. Frequency and amplitude discrimination along the kinesthetic-cutaneous continuum in the presence of masking stimuli. The Journal of the Acoustical Society of America 120, 5 (2006), 2789–2800.Google ScholarCross Ref
    33. Lynette A Jones and Ian W Hunter. 1990. A perceptual analysis of stiffness. Experimental Brain Research 79, 1 (1990), 150–156.Google ScholarCross Ref
    34. Dean Karnopp. 1985. Computer Simulation of Stick-Slip Friction in Mechanical Dynamic Systems. ASME Journal of dynamic Systems, Measurement and Control 107, 1 (Mar 1985), 100–103.Google Scholar
    35. Danny M. Kaufman, Timothy Edmunds, and Dinesh K. Pai. 2005. Fast Frictional Dynamics for Rigid Bodies. ACM Trans. Graph. 24, 3 (July 2005), 946–956. Google ScholarDigital Library
    36. Danny M. Kaufman, Shinjiro Sueda, Doug L. James, and Dinesh K. Pai. 2008. Staggered Projections for Frictional Contact in Multibody Systems. ACM Trans. Graph. 27, 5, Article 164 (Dec. 2008), 11 pages. Google ScholarDigital Library
    37. Seung-Chan Kim, Ali Israr, and Ivan Poupyrev. 2013. Tactile Rendering of 3D Features on Touch Surfaces. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST ’13). ACM, New York, NY, USA, 531–538. Google ScholarDigital Library
    38. Roberta L. Klatzky and Susan J. Lederman. 2002. Perceiving texture through a probe. Touch in virtual environments (2002), 180–193.Google Scholar
    39. Roberta L. Klatzky and Susan J. Lederman. 2008. Perceiving object properties through a rigid link. Haptic Rendering: Algorithms and Applications 1 (2008), 7–19.Google ScholarCross Ref
    40. Roberta L. Klatzky, Susan J. Lederman, Cheryl Hamilton, Molly Grindley, and Robert H. Swendsen. 2003. Feeling textures through a probe: Effects of probe and surface geometry and exploratory factors. Attention, Perception, & Psychophysics 65, 4 (2003), 613–631.Google ScholarCross Ref
    41. Johnny C. Lee, Paul H. Dietz, Darren Leigh, William S. Yerazunis, and Scott E. Hudson. 2004. Haptic Pen: A Tactile Feedback Stylus for Touch Screens. In Proceedings of the 17th Annual ACM Symposium on User Interface Software and Technology (UIST ’04). ACM, New York, NY, USA, 291–294. Google ScholarDigital Library
    42. Dingzeyu Li, David I. W. Levin, Wojciech Matusik, and Changxi Zheng. 2016. Acoustic Voxels: Computational Optimization of Modular Acoustic Filters. ACM Trans. Graph. 35, 4, Article 88 (July 2016), 12 pages. Google ScholarDigital Library
    43. Thomas H. Massie, J. Kenneth Salisbury, et al. 1994. The phantom haptic interface: A device for probing virtual objects. In Proceedings of the ASME winter annual meeting, symposium on haptic interfaces for virtual environment and teleoperator systems, Vol. 55. Chicago, IL, 295–300.Google Scholar
    44. Ralph Mayer. 1991. Artist’s Handbook of Materials and Techniques: Revised and Updated. Viking Books.Google Scholar
    45. Dominik L. Michels, Gerrit A. Sobottka, and Andreas G. Weber. 2014. Exponential Integrators for Stiff Elastodynamic Problems. ACM Trans. Graph. 33, 1, Article 7 (Feb. 2014), 20 pages. Google ScholarDigital Library
    46. Brian Mirtich and John Canny. 1995. Impulse-based Simulation of Rigid Bodies. In Proceedings of the 1995 Symposium on Interactive 3D Graphics (I3D ’95). ACM, New York, NY, USA, 181–ff. Google ScholarDigital Library
    47. Ilana Nisky, Assaf Pressman, Carla M. Pugh, Ferdinando A. Mussa-Ivaldi, and Amir Karniel. 2011. Perception and Action in Teleoperated Needle Insertion. EEE Trans. Haptics 4, 3 (July 2011), 155–166. Google ScholarDigital Library
    48. Miguel A. Otaduy and Ming C. Lin. 2004. A Perceptually-inspired Force Model for Haptic Texture Rendering. In Proceedings of the 1st Symposium on Applied Perception in Graphics and Visualization (APGV ’04). ACM, New York, NY, USA, 123–126. Google ScholarDigital Library
    49. Fabio Pellacini, James A. Ferwerda, and Donald P. Greenberg. 2000. Toward a psychophysically-based light reflection model for image synthesis. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques. ACM Press/Addison-Wesley Publishing Co., 55–64. Google ScholarDigital Library
    50. Alvaro G. Perez, Daniel Lobo, Francesco Chinello, Gabriel Cirio, Monica Malvezzi, Jose San Martin, Domenico Prattichizzo, and Miguel A. Otaduy. 2017. Optimization-based wearable tactile rendering. IEEE transactions on haptics (2017).Google Scholar
    51. Henry Petroski. 1992. The pencil: A history of design and circumstance. Alfred a Knopf Incorporated.Google Scholar
    52. Michal Piovarči, David I.W. Levin, Jason Rebello, Desai Chen, Roman Ďurikovič, Hanspeter Pfister, Wojciech Matusik, and Piotr Didyk. 2016. An Interaction-Aware, Perceptual Model For Non-Linear Elastic Objects. ACM Transactions on Graphics (Proc. SIGGRAPH) 35, 4 (2016). Google ScholarDigital Library
    53. Helena Pongrac. 2008. Vibrotactile perception: examining the coding of vibrations and the just noticeable difference under various conditions. Multimedia Systems 13, 4 (2008), 297–307. Google ScholarDigital Library
    54. Ivan Poupyrev, Makoto Okabe, and Shigeaki Maruyama. 2004. Haptic Feedback for Pen Computing: Directions and Strategies. In CHI ’04 Extended Abstracts on Human Factors in Computing Systems (CHI EA ’04). ACM, New York, NY, USA, 1309–1312. Google ScholarDigital Library
    55. Joseph M. Romano and Katherine J. Kuchenbecker. 2012. Creating Realistic Virtual Textures from Contact Acceleration Data. IEEE Transactions on Haptics 5, 2 (April 2012), 109–119. Google ScholarDigital Library
    56. Leonid Sigal, Moshe Mahler, Spencer Diaz, Kyna McIntosh, Elizabeth Carter, Timothy Richards, and Jessica Hodgins. 2015. A Perceptual Control Space for Garment Simulation. ACM Trans. Graph. 34, 4, Article 117 (July 2015), 10 pages. Google ScholarDigital Library
    57. D. Amnon Silverstein and Joyce E. Farrell. 2001. Efficient method for paired comparison. Journal of Electronic Imaging 10, 2 (2001), 394–398.Google ScholarCross Ref
    58. Hyunyoung Song, Hrvoje Benko, Francois Guimbretiere, Shahram Izadi, Xiang Cao, and Ken Hinckley. 2011. Grips and Gestures on a Multi-Touch Pen. 1323–1332. https://www.microsoft.com/en-us/research/publication/grips-and-gestures-on-a-multi-touch-pen/ Google ScholarDigital Library
    59. Gwidon W Stachowiak. 2006. Wear: materials, mechanisms and practice. John Wiley & Sons.Google Scholar
    60. Luis L. Thurstone. 1927. A law of comparative judgment. Psychological Review 34, 4 (1927).Google ScholarCross Ref
    61. Wouter M Bergmann Tiest and Astrid ML Kappers. 2009. Cues for haptic perception of compliance. Haptics, IEEE Transactions on 2, 4 (2009), 189–199. Google ScholarDigital Library
    62. Stephen Van Dulken. 2002. Inventing the 20th Century: 100 Inventions that Shaped the World from the Airplane to the Zipper. NYU Press.Google Scholar
    63. Hector Vazquez-Leal, Roberto Castaneda-Sheissa, Uriel Filobello-Nino, Arturo Sarmiento-Reyes, and Jesus Sanchez Orea. 2012. High Accurate Simple Approximation of Normal Distribution Integral. Mathematical Problems in Engineering 2012 (2012).Google Scholar
    64. Qinglong Wang, Xiangshi Ren, Sayan Sarcar, and Xiaoying Sun. 2016. EV-Pen: Leveraging Electrovibration Haptic Feedback in Pen Interaction. In Proceedings of the 2016 ACM on Interactive Surfaces and Spaces (ISS ’16). ACM, New York, NY, USA, 57–66. Google ScholarDigital Library
    65. Rhiannon Williams. 2015. Jony Ive interview: The story of the Apple Pencil. http://www.telegraph.co.uk/technology/apple/11988396/Jony-Ive-interview-The-story-of-the-Apple-Pencil.html. (2015). accessed on: 8 January 2018.Google Scholar
    66. Josh Wills, Sameer Agarwal, David Kriegman, and Serge Belongie. 2009. Toward a perceptual space for gloss. ACM Transactions on Graphics 28, 4 (2009), 1–15. Google ScholarDigital Library
    67. Takashi Yoshioka, Sliman Bensmaïa, James C. Craig, and Shen-Fu. Hsiao. 2007. Texture perception through direct and indirect touch: An analysis of perceptual space for tactile textures in two modes of exploration. Somatosensory & Motor Research 24, 1–2 (2007), 53–70.Google ScholarCross Ref
    68. Takashi Yoshioka and Julia Zhou. 2009. Factors Involved in Tactile Texture Perception Through Probes. Advanced Robotics 23, 6 (2009), 747–766.Google ScholarCross Ref

ACM Digital Library Publication:

Overview Page: