“Deep multispectral painting reproduction via multi-layer, custom-ink printing” – ACM SIGGRAPH HISTORY ARCHIVES

“Deep multispectral painting reproduction via multi-layer, custom-ink printing”

  • 2018 SA Technical Papers_Shi_Deep multispectral painting reproduction via multi-layer, custom-ink printing

Conference:


Type(s):


Title:

    Deep multispectral painting reproduction via multi-layer, custom-ink printing

Session/Category Title:   IM-material

Presenter(s)/Author(s):


Moderator(s):


Abstract:


    We propose a workflow for spectral reproduction of paintings, which captures a painting’s spectral color, invariant to illumination, and reproduces it using multi-material 3D printing. We take advantage of the current 3D printers’ capabilities of combining highly concentrated inks with a large number of layers, to expand the spectral gamut of a set of inks. We use a data-driven method to both predict the spectrum of a printed ink stack and optimize for the stack layout that best matches a target spectrum. This bidirectional mapping is modeled using a pair of neural networks, which are optimized through a problem-specific multi-objective loss function. Our loss function helps find the best possible ink layout resulting in the balance between spectral reproduction and colorimetric accuracy under a multitude of illuminants. In addition, we introduce a novel spectral vector error diffusion algorithm based on combining color contoning and halftoning, which simultaneously solves the layout discretization and color quantization problems, accurately and efficiently. Our workflow outperforms the state-of-the-art models for spectral prediction and layout optimization. We demonstrate reproduction of a number of real paintings and historically important pigments using our prototype implementation that uses 10 custom inks with varying spectra and a resin-based 3D printer.

References:


    1. Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. TensorFlow: A System for Large-scale Machine Learning. In Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation (OSDI’16). USENIX Association, Berkeley, CA, USA, 265–283. http://dl.acm.org/citation.cfm?id=3026877.3026899 Google ScholarDigital Library
    2. Satoshi Abet and Gabriel Marcu. 1994. A neural network approach for RGB to YMCK color conversion. In TENCON’94. IEEE Region 10’s Ninth Annual International Conference. Theme: Frontiers of Computer Technology. Proceedings of 1994. IEEE, 6–9.Google ScholarCross Ref
    3. Azadeh Asadi. 2017. Freelance artist. http://www.azadehasadi.com/index.html. (2017). Accessed: 2018-08-15.Google Scholar
    4. Teun Baar, Hans Brettel, and Maria V Ortiz Segovia. 2015. Towards gloss control in fine art reproduction. In Measuring, Modeling, and Reproducing Material Appearance 2015, Vol. 9398. International Society for Optics and Photonics, 93980T.Google Scholar
    5. Teun Baar, Sepideh Samadzadegan, Philipp Urban, and Maria V Ortiz Segovia. 2016. Interrelation between gloss and texture perception of 2.5 D-printed surfaces. Electronic Imaging 2016, 9 (2016), 1–6.Google Scholar
    6. Vahid Babaei and Roger D Hersch. 2016. N-Ink printer characterization with barycentric subdivision. IEEE Transactions on Image Processing 25, 7 (2016), 3023–3031.Google ScholarDigital Library
    7. Vahid Babaei, Kiril Vidimče, Michael Foshey, Alexandre Kaspar, Piotr Didyk, and Wojciech Matusik. 2017. Color contoning for 3D printing. ACM Trans. Graph. (SIGGRAPH) 36 (2017). Google ScholarDigital Library
    8. Seung-Hwan Baek, Incheol Kim, Diego Gutierrez, and Min H Kim. 2017. Compact single-shot hyperspectral imaging using a prism. ACM Transactions on Graphics (TOG) 36, 6 (2017), 217. Google ScholarDigital Library
    9. Farhan Baqai, Je-Ho Lee, Jan P Allebach, et al. 2005. Digital color halftoning. Signal Processing Magazine, IEEE 22, 1 (2005), 87–96.Google ScholarCross Ref
    10. Fausto Bernardini, Joshua Mittleman, Holly Rushmeier, Cláudio Silva, and Gabriel Taubin. 1999. The ball-pivoting algorithm for surface reconstruction. Visualization and Computer Graphics, IEEE Transactions on 5, 4 (1999), 349–359. Google ScholarDigital Library
    11. Roy S. Berns, Brittany D. Cox, and Farhad Moghareh Abed. 2015. Wavelength-dependent spatial correction and spectral calibration of a liquid crystal tunable filter imaging system. Appl. Opt. 54, 12 (Apr 2015), 3687–3693.Google ScholarCross Ref
    12. Roy S Berns, Lawrence A Taplin, Philipp Urban, and Yonghui Zhao. 2008. Spectral color reproduction of paintings. In Conference on Colour in Graphics, Imaging, and Vision, Vol. 2008. Society for Imaging Science and Technology, 484–488.Google Scholar
    13. Jean-Yves Bouguet. 2008. A Release of a Camera Calibration Toolbox for Matlab. (2008).Google Scholar
    14. Matthew Brown and David G Lowe. 2007. Automatic panoramic image stitching using invariant features. International journal of computer vision 74, 1 (2007), 59–73. Google ScholarDigital Library
    15. Alan Brunton, Can Ates Arikan, Tejas Madan Tanksale, and Philipp Urban. 2018. 3D Printing Spatially Varying Color and Translucency. ACM Trans. Graph. 37, 4, Article 157 (July 2018), 13 pages. Google ScholarDigital Library
    16. Alan Brunton, Can Ates Arikan, and Philipp Urban. 2015. Pushing the limits of 3d color printing: Error diffusion with translucent materials. ACM Transactions on Graphics (TOG) 35, 1 (2015), 4. Google ScholarDigital Library
    17. Yongda Chen, Roy S Berns, and Lawrence A Taplin. 2004. Six color printer characterization using an optimized cellular Yule-Nielsen spectral Neugebauer model. Journal of Imaging Science and Technology 48, 6 (2004), 519–528.Google Scholar
    18. Inchang Choi, Daniel S Jeon, Giljoo Nam, Diego Gutierrez, and Min H Kim. 2017. High-quality hyperspectral reconstruction using a spectral prior. ACM Transactions on Graphics (TOG) 36, 6 (2017), 218. Google ScholarDigital Library
    19. Antonino Cosentino. 2014. FORS Spectral Database of Historical Pigments in Different Binders. 2 (09 2014), 57–68.Google Scholar
    20. George Cybenko. 1989. Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals and Systems 2, 4 (01 Dec 1989), 303–314.Google Scholar
    21. Hugh R. Davidson and Henry Hemmendinger. 1966. Color Prediction Using the Two-Constant Turbid-Media Theory*. J. Opt. Soc. Am. 56, 8 (Aug 1966), 1102–1109.Google ScholarCross Ref
    22. Maxim Derhak and Mitchell Rosen. 2006. Spectral colorimetry using LabPQR: an interim connection space. Journal of Imaging Science and Technology 50, 1 (2006), 53–63.Google ScholarCross Ref
    23. Paul Apostolos Drakopoulos and Ganesh Subbarayan. 2002. Color printer characterization using optimization theory and neural networks. (Nov. 12 2002). US Patent 6,480,299.Google Scholar
    24. Oskar Elek, Denis Sumin, Ran Zhang, Tim Weyrich, Karol Myszkowski, Bernd Bickel, Alexander Wilkie, and Jaroslav Křivánek. 2017. Scattering-aware Texture Reproduction for 3D Printing. ACM Trans. Graph. 36, 6, Article 241 (Nov. 2017), 15 pages. Google ScholarDigital Library
    25. Willemijn S Elkhuizen, Tessa TW Essers, Boris Lenseigne, Clemens Weijkamp, Yu Song, Sylvia C Pont, Jo M-P Geraedts, and Joris Dik. 2017. Reproduction of Gloss, Color and Relief of Paintings using 3D Scanning and 3D Printing. (2017).Google Scholar
    26. Willemijn S Elkhuizen, Boris AJ Lenseigne, Teun Baar, Wim Verhofstad, Erik Tempelman, Jo MP Geraedts, and Joris Dik. 2015. Reproducing oil paint gloss in print for the purpose of creating reproductions of Old Masters. In Measuring, Modeling, and Reproducing Material Appearance 2015, Vol. 9398. International Society for Optics and Photonics, 93980W.Google Scholar
    27. Willemijn S Elkhuizen, Tim Zaman, Wim Verhofstad, Pieter P Jonker, Joris Dik, and Jo MP Geraedts. 2014. Topographical scanning and reproduction of near-planar surfaces of paintings. In Measuring, Modeling, and Reproducing Material Appearance, Vol. 9018. International Society for Optics and Photonics, 901809.Google Scholar
    28. Scott E Fahlman and Christian Lebiere. 1990. The cascade-correlation learning architecture. In Advances in neural information processing systems. 524–532. Google ScholarDigital Library
    29. Robert W. Floyd and Louis Steinberg. 1976. An Adaptive Algorithm for Spatial Greyscale. In Proceedings of Society for Information Displays, Vol. 17. 75–77.Google Scholar
    30. Jérémie Gerhardt and Jon Y Hardeberg. 2007. Controlling the error in spectral vector error diffusion. In Color Imaging XII: Processing, Hardcopy, and Applications, Vol. 6493. International Society for Optics and Photonics, 649316.Google Scholar
    31. Ioannis Gkioulekas, Shuang Zhao, Kavita Bala, Todd Zickler, and Anat Levin. 2013. Inverse Volume Rendering with Material Dictionaries. ACM Trans. Graph. 32, 6, Article 162 (Nov. 2013), 13 pages. Google ScholarDigital Library
    32. Jean Hergel and Sylvain Lefebvre. 2014. Clean color: Improving multi-filament 3D prints. In Computer Graphics Forum, Vol. 33. Wiley Online Library, 469–478. Google ScholarDigital Library
    33. Francisco H. Imai, Mitchell R. Rosen, and Roy S. Berns. 2002. Comparative study of metrics for spectral match quality. In Conference on Colour in Graphics, Imaging, and Vision, Vol. 2002. Society for Imaging Science and Technology, 492–496.Google Scholar
    34. Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. 2017. Image-to-Image Translation with Conditional Adversarial Networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21–26, 2017. 5967–5976.Google Scholar
    35. Wenzel Jakob, Eugene d’Eon, Otto Jakob, and Steve Marschner. 2014. A comprehensive framework for rendering layered materials. ACM Transactions on Graphics (ToG) 33, 4 (2014), 118. Google ScholarDigital Library
    36. Henry R Kang and Peter G Anderson. 1992. Neural network applications to the color scanner and printer calibrations. Journal of Electronic Imaging 1, 2 (1992), 125–136.Google ScholarCross Ref
    37. Isaac Kauvar, Samuel J Yang, Liang Shi, Ian McDowall, and Gordon Wetzstein. 2015. Adaptive color display via perceptually-driven factored spectral projection. ACM Transactions on Graphics (TOG) 34, 6 (2015), 165. Google ScholarDigital Library
    38. Takayuki Kawaguchi, Norimichi Tsumura, Hideaki Haneishi, Yoichi Miyake, and M Kouzaki. 1999. Vector error diffusion method for spectral color reproduction. In PICS. 394–397.Google Scholar
    39. Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).Google Scholar
    40. R. Victor Klassen, Reiner Eschbach, and Krishna Bharat. 1994. Vector error diffusion in a distorted colour space. In Proc. of IS&T 47th Annual Conference. 63–65.Google Scholar
    41. P. Kubelka and F. Munk. 1931. Ein Beitrag zur Optik der Farbanstriche. Zeitschrift für technische Physik 12 (1931), 593–601.Google Scholar
    42. D. J. Littlewood, P. A. Drakopoulos, and G. Subbarayan. 2002. Pareto-optimal Formulations for Cost Versus Colorimetric Accuracy Trade-offs in Printer Color Management. ACM Trans. Graph. 21, 2 (April 2002), 132–175. Google ScholarDigital Library
    43. M Ronnier Luo, Guihua Cui, and B Rigg. 2001. The development of the CIE 2000 colour-difference formula: CIEDE2000. Color Research & Application 26, 5 (2001), 340–350.Google ScholarCross Ref
    44. Gabriel Marcu and Kansei Iwata. 1993. RGB-YMCK color conversion by application of the neural networks. In Color and Imaging Conference, Vol. 1993. Society for Imaging Science and Technology, 27–32.Google Scholar
    45. Panagiotis Takis Metaxas. 1998. Optimal parallel error diffusion dithering. In Color Imaging: Device-Independent Color, Color Hardcopy, and Graphic Arts IV, Vol. 3648. International Society for Optics and Photonics, 485–495.Google Scholar
    46. Peter Morovič, Ján Morovič, Jordi Arnabat, and Juan Manuel García-Reyero. 2012. Revisiting spectral printing: A data driven approach. In Color and Imaging Conference, Vol. 2012. Society for Imaging Science and Technology, 335–340.Google Scholar
    47. Ole Norberg and Daniel Nyström. 2013. Extending color primary set in spectral vector error diffusion by multilevel halftoning. In Color Imaging XVIII: Displaying, Processing, Hardcopy, and Applications, Vol. 8652. International Society for Optics and Photonics, 86520M.Google Scholar
    48. Victor Ostromoukhov. 1993. Chromaticity gamut enhancement by heptatone multicolor printing. In IST/SPIE 1993 Symposium of Electronic Imaging: Science and Technology, Conf. on Device Independent Color Imaging and Imaging Systems Integration, Vol. 1905. 139–151.Google Scholar
    49. Daniele Panozzo, Olga Diamanti, Sylvain Paris, Marco Tarini, Evgeni Sorkine, and Olga Sorkine-Hornung. 2015. Texture Mapping Real-World Objects with Hydrographics. Computer Graphics Forum (proceedings of EUROGRAPHICS Symposium on Geometry Processing) 34, 5 (2015), 65–75. Google ScholarDigital Library
    50. Théo Phan Van Song, Christine Andraud, and Maria V Ortiz Segovia. 2016a. Implementation of the four-flux model for spectral and color prediction of 2.5 D prints. In NIP & Digital Fabrication Conference, Vol. 2016. Society for Imaging Science and Technology, 26–30.Google Scholar
    51. Théo Phan Van Song, Christine Andraud, and Maria V Ortiz-Segovia. 2016b. Towards spectral prediction of 2.5 D prints for soft-proofing applications. In Image Processing Theory Tools and Applications (IPTA), 2016 6th International Conference on. IEEE, 1–6.Google ScholarCross Ref
    52. Théo Phan Van Song, Christine Andraud, and Maria V Ortiz-Segovia. 2017. Spectral predictions of rough ink layers using a four-flux model. In Color and Imaging Conference, Vol. 2017. Society for Imaging Science and Technology, 251–257.Google Scholar
    53. Tim Reiner, Nathan Carr, Radomír Měch, Ondřej Št’ava, Carsten Dachsbacher, and Gavin Miller. 2014. Dual-color mixing for fused deposition modeling printers. In Computer Graphics Forum, Vol. 33. Wiley Online Library, 479–486. Google ScholarDigital Library
    54. Mitchell R Rosen and Maxim W Derhak. 2006. Spectral gamuts and spectral gamut mapping. In Spectral Imaging: Eighth International Symposium on Multispectral Color Science, Vol. 6062. International Society for Optics and Photonics, 60620K.Google ScholarCross Ref
    55. Mitchell R Rosen, Edward F Hattenberger, and Noboru Ohta. 2004. Spectral redundancy in a six-ink ink jet printer. Journal of Imaging Science and Technology 48, 3 (2004), 194–202.Google Scholar
    56. Romain Rossier. 2013. Framework for Printing with Daylight Fluorescent Inks. Ph.D. Dissertation. Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne.Google Scholar
    57. Claude Rozé, Thierry Girasole, and Anne-Gaelle Tafforin. 2001. Multilayer four-flux model of scattering, emitting and absorbing media. Atmospheric environment 35, 30 (2001), 5125–5130.Google Scholar
    58. Christian Schüller, Daniele Panozzo, Anselm Grundhöfer, Henning Zimmer, Evgeni Sorkine, and Olga Sorkine-Hornung. 2016. Computational thermoforming. ACM Transactions on Graphics (TOG) 35, 4 (2016), 43. Google ScholarDigital Library
    59. Pitchaya Sitthi-Amorn, Javier E. Ramos, Yuwang Wangy, Joyce Kwan, Justin Lan, Wenshou Wang, and Wojciech Matusik. 2015. MultiFab: A Machine Vision Assisted Platform for Multi-material 3D Printing. ACM Trans. Graph. 34, 4, Article 129 (July 2015), 11 pages. Google ScholarDigital Library
    60. Chang-Hwan Son, Hyung-Min Park, and Yeong-Ho Ha. 2011. Improved color separation based on dot-visibility modeling and color mixing rule for six-color printers. Journal of Imaging Science and Technology 55, 1 (2011), 10505–1.Google ScholarCross Ref
    61. Eric J Stollnitz, Victor Ostromoukhov, and David H Salesin. 1998. Reproducing color images using custom inks. In Proceedings of the 25th annual conference on Computer graphics and interactive techniques. ACM, 267–274. Google ScholarDigital Library
    62. Stratasys. 2016. Stratasys J750 the ultimate full-color multi-material 3D printer. http://www.stratasys.com/3d-printers/production-series/stratasys-j750. (2016). {Online; accessed 15-October-2016}.Google Scholar
    63. 3D Systems. 2013. ProJet 5500X Multi-Material 3D Printer. (2013). http://www.3dsystems.com/files/projet-5500x-1113-usen-web.pdfGoogle Scholar
    64. Lawrence A Taplin and Roy S Berns. 2001. Spectral color reproduction based on a six-color inkjet output system. In Color and Imaging Conference, Vol. 2001. Society for Imaging Science and Technology, 209–213.Google Scholar
    65. Shoji Tominaga. 1996. Color control using neural networks and its application. In Color Imaging: Device-Independent Color, Color Hard Copy, and Graphic Arts, Vol. 2658. International Society for Optics and Photonics, 253–261.Google Scholar
    66. Shoji Tominaga. 1998. Control scheme for printers using more than three color inks. In Electronic Imaging: Processing, Printing, and Publishing in Color, Vol. 3409. International Society for Optics and Photonics, 286–294.Google Scholar
    67. Shohei Tsutsumi, Mitchell Rosen, and Roy Berns. 2008. Spectral color reproduction using an interim connection space-based lookup table. Journal of Imaging Science and Technology 52, 4 (2008), 40201–1.Google ScholarCross Ref
    68. Philipp Urban and Roy S Berns. 2011. Paramer mismatch-based spectral gamut mapping. IEEE transactions on image processing 20, 6 (2011), 1599–1610. Google ScholarDigital Library
    69. Gunter Wyszecki and Walter Stanley Stiles. 1982. Color Science. Vol. 8. Wiley New York.Google Scholar
    70. Songhua Xu, Haisheng Tan, Xiantao Jiao, Francis Lau, and Yunhe Pan. 2007. A generic pigment model for digital painting. In Computer Graphics Forum, Vol. 26. Wiley Online Library, 609–618.Google Scholar
    71. Yizhong Zhang, Yiying Tong, and Kun Zhou. 2016. Coloring 3D Printed Surfaces by Thermoforming. IEEE Transactions on Visualization and Computer Graphics (2016).Google Scholar
    72. Yizhong Zhang, Chunji Yin, Changxi Zheng, and Kun Zhou. 2015. Computational hydrographic printing. ACM Transactions on Graphics (TOG) 34, 4 (2015), 131. Google ScholarDigital Library

ACM Digital Library Publication:


Overview Page:


Submit a story:

If you would like to submit a story about this presentation, please contact us: historyarchives@siggraph.org