“Paparazzi: surface editing by way of multi-view image processing”
Conference:
Type(s):
Title:
- Paparazzi: surface editing by way of multi-view image processing
Session/Category Title: Acquiring and editing, geometry via RGB (D) images
Presenter(s)/Author(s):
Moderator(s):
Abstract:
The image processing pipeline boasts a wide variety of complex filters and effects. Translating an individual effect to operate on 3D surface geometry inevitably results in a bespoke algorithm. Instead, we propose a general-purpose back-end optimization that allows users to edit an input 3D surface by simply selecting an off-the-shelf image processing filter. We achieve this by constructing a differentiable triangle mesh renderer, with which we can back propagate changes in the image domain to the 3D mesh vertex positions. The given image processing technique is applied to the entire shape via stochastic snapshots of the shape: hence, we call our method Paparazzi. We provide simple yet important design considerations to construct the Paparazzi renderer and optimization algorithms. The power of this rendering-based surface editing is demonstrated via the variety of image processing filters we apply. Each application uses an off-the-shelf implementation of an image processing method without requiring modification to the core Paparazzi algorithm.
References:
1. Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk. 2012. SLIC superpixels compared to state-of-the-art superpixel methods. IEEE transactions on pattern analysis and machine intelligence 34, 11 (2012), 2274–2282. Google ScholarDigital Library
2. Marco Attene. 2010. A lightweight approach to repairing digitized polygon meshes. The visual computer 26, 11 (2010), 1393–1406. Google ScholarDigital Library
3. Marco Attene. 2016. As-exact-as-possible repair of unprintable STL files. arXiv preprint arXiv:1605.07829 (2016).Google Scholar
4. Gavin Barill, Neil Dickson, Ryan Schmidt, David I.W. Levin, and Alec Jacobson. 2018. Fast Winding Numbers for Soups and Clouds. ACM Transactions on Graphics (2018). Google ScholarDigital Library
5. Jonathan T Barron and Jitendra Malik. 2015. Shape, illumination, and reflectance from shading. IEEE transactions on pattern analysis and machine intelligence 37, 8 (2015), 1670–1687.Google Scholar
6. Léon Bottou, Frank E Curtis, and Jorge Nocedal. 2016. Optimization methods for large-scale machine learning. arXiv preprint arXiv:1606.04838 (2016).Google Scholar
7. Tyson Brochu and Robert Bridson. 2009. Robust topological operations for dynamic explicit surfaces. SIAM Journal on Scientific Computing 31, 4 (2009), 2472–2493. Google ScholarDigital Library
8. Gyeongmin Choe, Jaesik Park, Yu-Wing Tai, and In So Kweon. 2017. Refining Geometry from Depth Sensors using IR Shading Images. Inter. Journal of Computer Vision (2017).Google Scholar
9. Amaël Delaunoy and Emmanuel Prados. 2011. Gradient flows for optimizing triangular mesh-based surfaces: Applications to 3d reconstruction problems dealing with visibility. International journal of computer vision 95, 2 (2011), 100–123. Google ScholarDigital Library
10. Timothy Dozat. 2016. Incorporating nesterov momentum into adam. (2016).Google Scholar
11. Jérémie Dumas, An Lu, Sylvain Lefebvre, Jun Wu, and Christian Dick. 2015. By-example synthesis of structurally sound patterns. ACM Transactions on Graphics (TOG) 34, 4 (2015), 137. Google ScholarDigital Library
12. Jihad El-Sana, Elvir Azanli, and Amitabh Varshney. 1999. Skip strips: maintaining triangle strips for view-dependent rendering. In Proceedings of the conference on Visualization’99: celebrating ten years. IEEE Computer Society Press, 131–138. Google ScholarDigital Library
13. Pau Gargallo, Emmanuel Prados, and Peter Sturm. 2007. Minimizing the reprojection error in surface reconstruction from images. In Computer Vision, 2007. ICCV 2007. IEEE 11th International Conference on. IEEE, 1–8.Google ScholarCross Ref
14. Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. 2016. Image Style Transfer Using Convolutional Neural Networks. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27–30, 2016. 2414–2423.Google Scholar
15. Kyle Genova, Forrester Cole, Aaron Maschinot, Aaron Sarna, Daniel Vlasic, and William T. Freeman. 2018. Unsupervised Training for 3D Morphable Model Regression. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
16. Yotam Gingold and Denis Zorin. 2008. Shading-based surface editing. ACM Transactions on Graphics (TOG) 27, 3 (2008), 95. Google ScholarDigital Library
17. Ioannis Gkioulekas, Shuang Zhao, Kavita Bala, Todd Zickler, and Anat Levin. 2013. Inverse volume rendering with material dictionaries. ACM Transactions on Graphics (TOG) 32, 6 (2013), 162. Google ScholarDigital Library
18. Henri Gouraud. 1971. Continuous shading of curved surfaces. IEEE transactions on computers 100, 6 (1971), 623–629. Google ScholarDigital Library
19. Xianfeng Gu, Steven J Gortler, and Hugues Hoppe. 2002. Geometry images. ACM Transactions on Graphics (TOG) 21, 3 (2002), 355–361. Google ScholarDigital Library
20. Kaiming He and Jian Sun. 2015. Fast Guided Filter. CoRR abs/1505.00996 (2015). http://arxiv.org/abs/1505.00996Google Scholar
21. Kaiming He, Jian Sun, and Xiaoou Tang. 2010. Guided image filtering. In European conference on computer vision. Springer, 1–14.Google ScholarDigital Library
22. Lei He and Scott Schaefer. 2013. Mesh denoising via L 0 minimization. ACM Transactions on Graphics (TOG) 32, 4 (2013), 64. Google ScholarDigital Library
23. Aaron Hertzmann, Charles E Jacobs, Nuria Oliver, Brian Curless, and David H Salesin. 2001. Image analogies. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques. ACM, 327–340. Google ScholarDigital Library
24. Hugues Hoppe. 1997. View-dependent refinement of progressive meshes. In Proceedings of the 24th annual conference on Computer graphics and interactive techniques. ACM Press/Addison-Wesley Publishing Co., 189–198. Google ScholarDigital Library
25. Alec Jacobson, Daniele Panozzo, et al. 2016. libigl: A simple C++ geometry processing library. (2016). http://libigl.github.io/libigl/.Google Scholar
26. Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. 2017. Neural 3D Mesh Renderer. CoRR abs/1711.07566 (2017). arXiv:1711.07566 http://arxiv.org/abs/1711.07566Google Scholar
27. Bertrand Kerautret, Xavier Granier, and Achille Braquelaire. 2005. Intuitive shape modeling by shading design. In International Symposium on Smart Graphics. Springer, 163–174. Google ScholarDigital Library
28. Felix Knöppel, Keenan Crane, Ulrich Pinkall, and Peter Schröder. 2015. Stripe Patterns on Surfaces. ACM Trans. Graph. (2015). Google ScholarDigital Library
29. Leif Kobbelt and Mario Botsch. 2004. A survey of point-based techniques in computer graphics. Computers & Graphics 28, 6 (2004), 801–814. Google ScholarDigital Library
30. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems. 1097–1105. Google ScholarDigital Library
31. Y-K Lai, S-M Hu, DX Gu, and Ralph R Martin. 2005. Geometric texture synthesis and transfer via geometry images. In Proc. SPM. Google ScholarDigital Library
32. Eric Landreneau and Scott Schaefer. 2010. Scales and Scale-like Structures. Comput. Graph. Forum (2010).Google Scholar
33. Peter Lindstrom and Greg Turk. 2000. Image-driven simplification. ACM Transactions on Graphics (ToG) 19, 3 (2000), 204–241. Google ScholarDigital Library
34. Baoxia Liu and James T Todd. 2004. Perceptual biases in the interpretation of 3D shape from shading. Vision research (2004).Google Scholar
35. Guilin Liu, Duygu Ceylan, Ersin Yumer, Jimei Yang, and Jyh-Ming Lien. 2017. Material Editing Using a Physically Based Rendering Network. In 2017 IEEE International Conference on Computer Vision (ICCV). IEEE, 2280–2288.Google ScholarCross Ref
36. Matthew M Loper and Michael J Black. 2014. OpenDR: An approximate differentiable renderer. In European Conference on Computer Vision. Springer, 154–169.Google ScholarCross Ref
37. David Luebke and Carl Erikson. 1997. View-dependent simplification of arbitrary polygonal environments. In Proceedings of the 24th annual conference on Computer graphics and interactive techniques. ACM Press/Addison-Wesley Publishing Co., 199–208. Google ScholarDigital Library
38. David Luebke and Benjamin Hallen. 2001. Perceptually driven simplification for interactive rendering. In Rendering Techniques 2001. Springer, 223–234. Google ScholarDigital Library
39. Stephen R. Marschner and Donald P. Greenberg. 1997. Inverse Lighting for Photography. In Color and Imaging Conference.Google Scholar
40. Roy Or-El, Rom Hershkovitz, Aaron Wetzler, Guy Rosman, Alfred M. Bruckstein, and Ron Kimmel. 2016. Real-Time Depth Refinement for Specular Objects. In Proc. CVPR.Google ScholarCross Ref
41. Celal Ozturk, Emrah Hancer, and Dervis Karaboga. 2014. Color image quantization: a short review and an application with artificial bee colony algorithm. Informatica 25, 3 (2014), 485–503. Google ScholarDigital Library
42. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch. (2017).Google Scholar
43. Bui Tuong Phong. 1975. Illumination for computer generated pictures. Commun. ACM 18, 6 (1975), 311–317. Google ScholarDigital Library
44. Emmanuel Prados and Olivier Faugeras. 2006. Shape from shading. Handbook of mathematical models in computer vision (2006), 375–388.Google Scholar
45. Ravi Ramamoorthi and Pat Hanrahan. 2001. A Signal-processing Framework for Inverse Rendering. In Proc. SIGGRAPH. Google ScholarDigital Library
46. Nadia Robertini, Dan Casas, Edilson Aguiar, and Christian Theobalt. 2017. Multi-view Performance Capture of Surface Details. Int. J. Comput. Vision (2017). Google ScholarDigital Library
47. Sebastian Ruder. 2016. An overview of gradient descent optimization algorithms. arXiv preprint arXiv:1609.04747 (2016).Google Scholar
48. Ryan Schmidt and Karan Singh. 2010. Meshmixer: an interface for rapid mesh composition. In ACM SIGGRAPH 2010 Talks. ACM, 6. Google ScholarDigital Library
49. Christian Schüller, Daniele Panozzo, and Olga Sorkine-Hornung. 2014. Appearance-mimicking Surfaces. ACM Trans. Graph. (2014). Google ScholarDigital Library
50. Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).Google Scholar
51. Kenshi Takayama, Ryan Schmidt, Karan Singh, Takeo Igarashi, Tamy Boubekeur, and Olga Sorkine. 2011. Geobrush: Interactive mesh geometry cloning. In Computer Graphics Forum, Vol. 30. Wiley Online Library, 613–622.Google Scholar
52. Elif Tosun, Yotam I Gingold, Jason Reisman, and Denis Zorin. 2007. Shape optimization using reflection lines. In Proc. SGP. Google ScholarDigital Library
53. Greg Turk. 1991. Generating textures on arbitrary surfaces using reaction-diffusion. Siggraph (1991). Google ScholarDigital Library
54. Stefan Van der Walt, Johannes L Schönberger, Juan Nunez-Iglesias, François Boulogne, Joshua D Warner, Neil Yager, Emmanuelle Gouillart, and Tony Yu. 2014. scikit-image: image processing in Python. PeerJ 2 (2014), e453.Google ScholarCross Ref
55. CWAM Van Overveld. 1996. Painting gradients: Free-form surface design using shading patterns.. In Graphics Interface, Vol. 96. 151–158. Google ScholarDigital Library
56. Li-Yi Wei and Marc Levoy. 2001. Texture synthesis over arbitrary manifold surfaces. Siggraph (2001). Google ScholarDigital Library
57. Martin Weier, Michael Stengel, Thorsten Roth, Piotr Didyk, Elmar Eisemann, Martin Eisemann, Steve Grogorick, André Hinkenjann, Ernst Kruijff, Marcus Magnor, et al. 2017. Perception-driven Accelerated Rendering. In Computer Graphics Forum, Vol. 36. Wiley Online Library, 611–643. Google ScholarDigital Library
58. Nathaniel Williams, David Luebke, Jonathan D Cohen, Michael Kelley, and Brenden Schubert. 2003. Perceptually guided simplification of lit, textured meshes. In Proceedings of the 2003 symposium on Interactive 3D graphics. ACM, 113–121. Google ScholarDigital Library
59. Robert J Woodham. 1980. Photometric method for determining surface orientation from multiple images. Optical engineering 19, 1 (1980), 191139.Google Scholar
60. Chenglei Wu, Kiran Varanasi, Yebin Liu, Hans-Peter Seidel, and Christian Theobalt. 2011. Shading-based Dynamic Shape Refinement from Multi-view Video Under General Illumination. In Proc. ICCV. Google ScholarDigital Library
61. Chenglei Wu, Michael Zollhöfer, Matthias Nießner, Marc Stamminger, Shahram Izadi, and Christian Theobalt. 2014. Real-time shading-based refinement for consumer depth cameras. ACM Transactions on Graphics (TOG) 33, 6 (2014), 200. Google ScholarDigital Library
62. Julie C Xia and Amitabh Varshney. 1996. Dynamic view-dependent simplification for polygonal models. In Proceedings of the 7th conference on Visualization’96. IEEE Computer Society Press, 327-ff. Google ScholarDigital Library
63. Li Xu, Cewu Lu, Yi Xu, and Jiaya Jia. 2011. Image smoothing via L 0 gradient minimization. In ACM Transactions on Graphics (TOG), Vol. 30. ACM, 174. Google ScholarDigital Library
64. Ruo Zhang, Ping-Sing Tsai, James Edwin Cryer, and Mubarak Shah. 1999. Shape-from-shading: a survey. IEEE transactions on pattern analysis and machine intelligence 21, 8 (1999), 690–706. Google ScholarDigital Library
65. Wangyu Zhang, Bailin Deng, Juyong Zhang, Sofien Bouaziz, and Ligang Liu. 2015. Guided mesh normal filtering. In Computer Graphics Forum, Vol. 34. Wiley Online Library, 23–34. Google ScholarDigital Library
66. Shuang Zhao. 2014. Modeling and rendering fabrics at micron-resolution. Cornell University.Google Scholar
