“Differentiable surface splatting for point-based geometry processing” by Wang, Serena, Wu, Öztireli and Sorkine-Hornung
Conference:
Type(s):
Title:
- Differentiable surface splatting for point-based geometry processing
Session/Category Title: Differentiable Rendering
Presenter(s)/Author(s):
Moderator(s):
Abstract:
We propose Differentiable Surface Splatting (DSS), a high-fidelity differentiable renderer for point clouds. Gradients for point locations and normals are carefully designed to handle discontinuities of the rendering function. Regularization terms are introduced to ensure uniform distribution of the points on the underlying surface. We demonstrate applications of DSS to inverse rendering for geometry synthesis and denoising, where large scale topological changes, as well as small scale detail modifications, are accurately and robustly handled without requiring explicit connectivity, outperforming state-of-the-art techniques. The data and code are at https://github.com/yifita/DSS.
References:
1. Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk. 2012. SLIC superpixels compared to state-of-the-art superpixel methods. IEEE transactions on pattern analysis and machine intelligence 34, 11 (2012), 2274–2282.Google ScholarDigital Library
2. Marc Alexa, Johannes Behr, Daniel Cohen-Or, Shachar Fleishman, David Levin, and Claudio T Silva. 2003. Computing and rendering point set surfaces. IEEE Trans. Visualization & Computer Graphics 9, 1 (2003), 3–15.Google ScholarDigital Library
3. Dejan Azinović, Tzu-Mao Li, Anton Kaplanyan, and Matthias Nießner. 2019. Inverse Path Tracing for Joint Material and Lighting Estimation. arXiv preprint arXiv:1903.07145 (2019).Google Scholar
4. Matthew Berger, Andrea Tagliasacchi, Lee M Seversky, Pierre Alliez, Gael Guennebaud, Joshua A Levine, Andrei Sharf, and Claudio T Silva. 2017. A survey of surface reconstruction from point clouds. In Computer Graphics Forum, Vol. 36. 301–329.Google ScholarDigital Library
5. Paolo Cignoni, Marco Callieri, Massimiliano Corsini, Matteo Dellepiane, Fabio Ganovelli, and Guido Ranzuglia. 2008. MeshLab: an Open-Source Mesh Processing Tool. In Eurographics Italian Chapter Conference.Google Scholar
6. Massimiliano Corsini, Paolo Cignoni, and Roberto Scopigno. 2012. Efficient and flexible sampling with blue noise properties of triangular meshes. IEEE Trans. Visualization & Computer Graphics 18, 6 (2012), 914–924.Google ScholarDigital Library
7. Haoqiang Fan, Hao Su, and Leonidas J Guibas. 2017. A Point Set Generation Network for 3D Object Reconstruction from a Single Image. Proc. IEEE Conf. on Computer Vision & Pattern Recognition 2, 4, 6.Google ScholarCross Ref
8. Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proc. Inter. Conf. on Artificial Intelligence and Statistics. 249–256.Google Scholar
9. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative Adversarial Nets. In In Advances in Neural Information Processing Systems (NIPS).Google Scholar
10. Paul Guerrero, Yanir Kleiman, Maks Ovsjanikov, and Niloy J Mitra. 2018. PCPNet learning local shape properties from raw point clouds. In Computer Graphics Forum (Proc. of Eurographics), Vol. 37. 75–85.Google ScholarCross Ref
11. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In Proc. IEEE Conf. on Computer Vision & Pattern Recognition.Google ScholarCross Ref
12. Paul S Heckbert. 1989. Fundamentals of texture mapping and image warping. (1989).Google Scholar
13. Pedro Hermosilla, Tobias Ritschel, and Timo Ropinski. 2019. Total Denoising: Unsupervised Learning of 3D Point Cloud Cleaning. arXiv preprint arXiv:1904.07615 (2019).Google Scholar
14. P. Hermosilla, T. Ritschel, P-P Vazquez, A. Vinacua, and T. Ropinski. 2018. Monte Carlo Convolution for Learning on Non-Uniformly Sampled Point Clouds. ACM Trans. on Graphics (Proc. of SIGGRAPH Asia) 37, 6 (2018).Google Scholar
15. Hugues Hoppe, Tony DeRose, Tom Duchamp, John McDonald, and Werner Stuetzle. 1992. Surface reconstruction from unorganized points. Proc. of SIGGRAPH (1992), 71–78.Google ScholarDigital Library
16. Tao Hu, Zhizhong Han, Abhinav Shrivastava, and Matthias Zwicker. 2019. Render4Completion: Synthesizing Multi-view Depth Maps for 3D Shape Completion. arXiv preprint arXiv:1904.08366 (2019).Google Scholar
17. Hui Huang, Dan Li, Hao Zhang, Uri Ascher, and Daniel Cohen-Or. 2009. Consolidation of Unorganized Point Clouds for Surface Reconstruction. ACM Trans. on Graphics (Proc. of SIGGRAPH Asia) 28, 5 (2009), 176:1–176:7.Google Scholar
18. Hui Huang, Shihao Wu, Minglun Gong, Daniel Cohen-Or, Uri Ascher, and Hao Richard Zhang. 2013. Edge-Aware Point Set Resampling. ACM Trans. on Graphics 32, 1 (2013), 9:1–9:12.Google ScholarDigital Library
19. Eldar Insafutdinov and Alexey Dosovitskiy. 2018. Unsupervised learning of shape and pose with differentiable point clouds. In In Advances in Neural Information Processing Systems (NIPS). 2802–2812.Google Scholar
20. Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. 2017. Image-To-Image Translation With Conditional Adversarial Networks. In Proc. IEEE Conf. on Computer Vision & Pattern Recognition.Google Scholar
21. Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. 2018. Progressive Growing of GANs for Improved Quality, Stability, and Variation. In Proc. Int. Conf. on Learning Representations.Google Scholar
22. Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. 2018. Neural 3d mesh renderer. In Proc. IEEE Conf. on Computer Vision & Pattern Recognition. 3907–3916.Google ScholarCross Ref
23. Fabian Langguth, Kalyan Sunkavalli, Sunil Hadap, and Michael Goesele. 2016. Shading-aware multi-view stereo. In Proc. Euro. Conf. on Computer Vision. Springer, 469–485.Google ScholarCross Ref
24. Tzu-Mao Li, Miika Aittala, Frédo Durand, and Jaakko Lehtinen. 2018. Differentiable monte carlo ray tracing through edge sampling. In ACM Trans. on Graphics (Proc. of SIGGRAPH Asia). ACM, 222.Google Scholar
25. Chen-Hsuan Lin, Chen Kong, and Simon Lucey. 2018. Learning efficient point cloud generation for dense 3D object reconstruction. In AAAI Conference on Artificial Intelligence.Google Scholar
26. Yaron Lipman, Daniel Cohen-Or, David Levin, and Hillel Tal-Ezer. 2007. Parameterization-free projection for geometry reconstruction. ACM Trans. on Graphics (Proc. of SIGGRAPH) 26, 3 (2007), 22:1–22:6.Google ScholarDigital Library
27. Guilin Liu, Duygu Ceylan, Ersin Yumer, Jimei Yang, and Jyh-Ming Lien. 2017. Material editing using a physically based rendering network. In Proc. IEEE Conf. on Computer Vision & Pattern Recognition. 2261–2269.Google ScholarCross Ref
28. Hsueh-Ti Derek Liu, Michael Tao, and Alec Jacobson. 2018. Paparazzi: Surface Editing by way of Multi-View Image Processing. In ACM Trans. on Graphics (Proc. of SIGGRAPH Asia). ACM, 221.Google Scholar
29. Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. 2019. Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning. arXiv preprint arXiv:1904.01786 (2019).Google Scholar
30. Matthew M Loper and Michael J Black. 2014. OpenDR: An approximate differentiable renderer. In Proc. Euro. Conf. on Computer Vision. Springer, 154–169.Google ScholarCross Ref
31. Xuequan Lu, Shihao Wu, Honghua Chen, Sai-Kit Yeung, Wenzhi Chen, and Matthias Zwicker. 2018. GPF: GMM-inspired feature-preserving point set filtering. IEEE Trans. Visualization & Computer Graphics 24, 8 (2018), 2315–2326.Google ScholarCross Ref
32. Robert Maier, Kihwan Kim, Daniel Cremers, Jan Kautz, and Matthias Nießner. 2017. Intrinsic3d: High-quality 3D reconstruction by joint appearance and geometry optimization with spatially-varying lighting. In Proc. Int. Conf. on Computer Vision. 3114–3122.Google ScholarCross Ref
33. Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. 2017. Least squares generative adversarial networks. In Proc. Int. Conf. on Computer Vision. 2794–2802.Google ScholarCross Ref
34. Thu H Nguyen-Phuoc, Chuan Li, Stephen Balaban, and Yongliang Yang. 2018. RenderNet: A deep convolutional network for differentiable rendering from 3D shapes. In In Advances in Neural Information Processing Systems (NIPS). 7891–7901.Google Scholar
35. A Cengiz Öztireli, Gael Guennebaud, and Markus Gross. 2009. Feature preserving point set surfaces based on non-linear kernel regression. In Computer Graphics Forum (Proc. of Eurographics), Vol. 28. 493–501.Google ScholarCross Ref
36. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch. In NIPS-W.Google Scholar
37. Felix Petersen, Amit H Bermano, Oliver Deussen, and Daniel Cohen-Or. 2019. Pix2Vex: Image-to-Geometry Reconstruction using a Smooth Differentiable Renderer. arXiv preprint arXiv:1903.11149 (2019).Google Scholar
38. Hanspeter Pfister, Matthias Zwicker, Jeroen Van Baar, and Markus Gross. 2000. Surfels: Surface elements as rendering primitives. In Proc. Conf. on Computer Graphics and Interactive techniques. 335–342.Google ScholarDigital Library
39. Jhony K Pontes, Chen Kong, Sridha Sridharan, Simon Lucey, Anders Eriksson, and Clinton Fookes. 2017. Image2Mesh: A Learning Framework for Single Image 3D Reconstruction. arXiv preprint arXiv:1711.10669 (2017).Google Scholar
40. Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. 2017a. PointNet: Deep learning on point sets for 3D classification and segmentation. In Proc. IEEE Conf. on Computer Vision & Pattern Recognition.Google Scholar
41. Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. 2017b. PointNet++: Deep hierarchical feature learning on point sets in a metric space. In In Advances in Neural Information Processing Systems (NIPS). 5099–5108.Google Scholar
42. Sai Rajeswar, Fahim Mannan, Florian Golemo, David Vazquez, Derek Nowrouzezahrai, and Aaron Courville. 2018. Pix2Scene: Learning Implicit 3D Representations from Images. (2018).Google Scholar
43. Marie-Julie Rakotosaona, Vittorio La Barbera, Paul Guerrero, Niloy J Mitra, and Maks Ovsjanikov. 2019. POINTCLEANNET: Learning to Denoise and Remove Outliers from Dense Point Clouds. arXiv preprint arXiv:1901.01060 (2019).Google Scholar
44. Elad Richardson, Matan Sela, Roy Or-El, and Ron Kimmel. 2017. Learning detailed face reconstruction from a single image. In IEEE Trans. Pattern Analysis & Machine Intelligence. 1259–1268.Google Scholar
45. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In Inter. Conf. on Medical image computing and computer-assisted intervention. Springer, 234–241.Google ScholarCross Ref
46. Riccardo Roveri, A Cengiz Öztireli, Ioana Pandele, and Markus Gross. 2018a. Pointpronets: Consolidation of point clouds with convolutional neural networks. In Computer Graphics Forum (Proc. of Eurographics), Vol. 37. 87–99.Google ScholarCross Ref
47. Riccardo Roveri, Lukas Rahmann, Cengiz Oztireli, and Markus Gross. 2018b. A network architecture for point cloud classification via automatic depth images generation. In Proc. IEEE Conf. on Computer Vision & Pattern Recognition. 4176–4184.Google ScholarCross Ref
48. Soumyadip Sengupta, Angjoo Kanazawa, Carlos D Castillo, and David W Jacobs. 2018. SfSNet: Learning Shape, Reflectance and Illuminance of Facesin the Wild’. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6296–6305.Google ScholarCross Ref
49. Jian Shi, Yue Dong, Hao Su, and Stella X. Yu. 2017. Learning Non-Lambertian Object Intrinsics Across ShapeNet Categories. In Proc. IEEE Conf. on Computer Vision & Pattern Recognition.Google Scholar
50. Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhöfer. 2018. DeepVoxels: Learning Persistent 3D Feature Embeddings. arXiv preprint arXiv:1812.01024 (2018).Google Scholar
51. Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. 2013. On the importance of initialization and momentum in deep learning. In Proc. IEEE Int. Conf. on Machine Learning. 1139–1147.Google Scholar
52. Shubham Tulsiani, Tinghui Zhou, Alexei A Efros, and Jitendra Malik. 2017. Multi-view supervision for single-view reconstruction via differentiable ray consistency. In Proc. IEEE Conf. on Computer Vision & Pattern Recognition. 2626–2634.Google ScholarCross Ref
53. Thijs Vogels, Fabrice Rousselle, Brian McWilliams, Gerhard Röthlin, Alex Harvill, David Adler, Mark Meyer, and Jan Novák. 2018. Denoising with kernel prediction and asymmetric loss functions. ACM Trans. on Graphics 37, 4 (2018), 124.Google ScholarDigital Library
54. Peng-Shuai Wang, Yang Liu, and Xin Tong. 2016. Mesh denoising via cascaded normal regression. ACM Trans. on Graphics (Proc. of SIGGRAPH Asia) 35, 6 (2016), 232–1.Google Scholar
55. Li Xu, Cewu Lu, Yi Xu, and Jiaya Jia. 2011. Image smoothing via L 0 gradient minimization. In ACM Transactions on Graphics (TOG), Vol. 30. ACM, 174.Google ScholarDigital Library
56. Xinchen Yan, Jimei Yang, Ersin Yumer, Yijie Guo, and Honglak Lee. 2016. Perspective transformer nets: Learning single-view 3d object reconstruction without 3d supervision. In In Advances in Neural Information Processing Systems (NIPS). 1696–1704.Google Scholar
57. Shunyu Yao, Tzu Ming Hsu, Jun-Yan Zhu, Jiajun Wu, Antonio Torralba, Bill Freeman, and Josh Tenenbaum. 2018. 3D-aware scene manipulation via inverse graphics. In In Advances in Neural Information Processing Systems (NIPS). 1887–1898.Google Scholar
58. Wang Yifan, Shihao Wu, Hui Huang, Daniel Cohen-Or, and Olga Sorkine-Hornung. 2018. Patch-based Progressive 3D Point Set Upsampling. arXiv preprint arXiv:1811.11286 (2018).Google Scholar
59. Lequan Yu, Xianzhi Li, Chi-Wing Fu, Daniel Cohen-Or, and Pheng-Ann Heng. 2018. EC-Net: an Edge-aware Point set Consolidation Network. Proc. Euro. Conf. on Computer Vision (2018).Google ScholarCross Ref
60. Rui Zhu, Hamed Kiani Galoogahi, Chaoyang Wang, and Simon Lucey. 2017. Rethinking reprojection: Closing the loop for pose-aware shape reconstruction from a single image. In Proc. Int. Conf. on Computer Vision. 57–65.Google ScholarCross Ref
61. Matthias Zwicker, Mark Pauly, Oliver Knoll, and Markus Gross. 2002. Pointshop 3D: An interactive system for point-based surface editing. In ACM Trans. on Graphics, Vol. 21. ACM, 322–329.Google ScholarDigital Library
62. Matthias Zwicker, Hanspeter Pfister, Jeroen Van Baar, and Markus Gross. 2001. Surface splatting. In Proc. Conf. on Computer Graphics and Interactive techniques. ACM, 371–378.Google ScholarDigital Library
63. Matthias Zwicker, Jussi Räsänen, Mario Botsch, Carsten Dachsbacher, and Mark Pauly. 2004. Perspective accurate splatting. In Proc. of Graphics interface. Canadian Human-Computer Communications Society, 247–254.Google Scholar
