“Physics-informed Learning of Characteristic Trajectories for Smoke Reconstruction”
Conference:
Type(s):
Title:
- Physics-informed Learning of Characteristic Trajectories for Smoke Reconstruction
Presenter(s)/Author(s):
Abstract:
We introduce Neural Characteristic Trajectory Fields, a novel representation utilizing Eulerian neural fields to implicitly model Lagrangian fluid trajectories for video-based fluid reconstruction. This topology-free, auto-differentiable representation facilitates end-to-end supervision, encompassing long-term conservation and short-term physics priors. It offers advancements in high-fidelity fluid reconstruction across synthetic and real scenes.
References:
[1]
Bradley Atcheson, Ivo Ihrke, Wolfgang Heidrich, Art Tevs, Derek Bradley, Marcus Magnor, and Hans-Peter Seidel. 2008. Time-resolved 3d capture of non-stationary gas flows. ACM Trans. Graph. 27, 5, Article 132 (dec 2008), 9 pages. https://doi.org/10.1145/1409060.1409085
[2]
Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan. 2021. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 5855?5864.
[3]
Kai Berger, Ivo Ihrke, Bradley Atcheson, Wolfgang Heidrich, and Marcus Magnor. 2009. Tomographic 4D Reconstruction of Gas Flows in the Presence of Occluders.VMV 2009 – Proceedings of the Vision, Modeling, and Visualization Workshop 2009, 29?36.
[4]
K Bi, L Xie, H Zhang, X Chen, X Gu, and Q Tian. 2023. Accurate medium-range global weather forecasting with 3D neural networks.Nature 619, 7970 (2023), 533?538.
[5]
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. 2022. TensoRF: Tensorial Radiance Fields. In European Conference on Computer Vision (ECCV).
[6]
Honglin Chen, Rundi Wu, Eitan Grinspun, Changxi Zheng, and Peter Yichen Chen. 2023. Implicit neural spatial representations for time-dependent pdes. In International Conference on Machine Learning. PMLR, 5162?5177.
[7]
Mengyu Chu, Lingjie Liu, Quan Zheng, Erik Franz, Hans-Peter Seidel, Christian Theobalt, and Rhaleb Zayer. 2022. Physics informed neural fields for smoke reconstruction with sparse data. ACM Transactions on Graphics (TOG) 41, 4 (2022), 1?14.
[8]
Mengyu Chu, Nils Thuerey, Hans-Peter Seidel, Christian Theobalt, and Rhaleb Zayer. 2021. Learning meaningful controls for fluids. ACM Transactions on Graphics (TOG) 40, 4 (2021), 1?13.
[9]
Yitong Deng, Hong-Xing Yu, Jiajun Wu, and Bo Zhu. 2023a. Learning Vortex Dynamics for Fluid Inference and Prediction. In Proceedings of the International Conference on Learning Representations.
[10]
Yitong Deng, Hong-Xing Yu, Diyang Zhang, Jiajun Wu, and Bo Zhu. 2023b. Fluid Simulation on Neural Flow Maps. ACM Trans. Graph. 42, 6 (2023).
[11]
Xin Dong, Ruize Wu, Chao Xiong, Hai Li, Lei Cheng, Yong He, Shiyou Qian, Jian Cao, and Linjian Mo. 2022. GDOD: Effective Gradient Descent using Orthogonal Decomposition for Multi-Task Learning. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management. 386?395.
[12]
Yilun Du, Yinan Zhang, Hong-Xing Yu, Joshua B Tenenbaum, and Jiajun Wu. 2021. Neural radiance flow for 4d view synthesis and video processing. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE Computer Society, 14304?14314.
[13]
Marie-Lena Eckert, Kiwon Um, and Nils Thuerey. 2019. ScalarFlow: a large-scale volumetric data set of real-world scalar transport flows for computer animation and machine learning. ACM Transactions on Graphics (TOG) 38, 6 (2019), 1?16.
[14]
Gerrit E Elsinga, Fulvio Scarano, Bernhard Wieneke, and Bas W van Oudheusden. 2006. Tomographic particle image velocimetry. Experiments in fluids 41, 6 (2006), 933?947.
[15]
Erik Franz, Barbara Solenthaler, and Nils Thuerey. 2021. Global Transport for Fluid Reconstruction with Learned Self-Supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1632?1642.
[16]
Guy Gafni, Justus Thies, Michael Zollhoefer, and Matthias Niessner. 2020. Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction. arxiv:2012.03065 [cs.CV]
[17]
Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang. 2021. Dynamic View Synthesis from Dynamic Monocular Video. In Proceedings of the IEEE International Conference on Computer Vision.
[18]
I. Grant. 1997. Particle Image Velocimetry: a Review. Proceedings of the Institution of Mechanical Engineers 211, 1 (1997), 55?76.
[19]
James Gregson, Ivo Ihrke, Nils Thuerey, and Wolfgang Heidrich. 2014. From capture to simulation: connecting forward and inverse problems in fluids. ACM Transactions on Graphics (TOG) 33, 4 (2014), 1?11.
[20]
Jinwei Gu, S.K. Nayar, E. Grinspun, P.N. Belhumeur, and R. Ramamoorthi. 2013. Compressive Structured Light for Recovering Inhomogeneous Participating Media. PAMI 35, 3 (2013), 555?567.
[21]
Yu Ji, Jinwei Ye, and Jingyi Yu. 2013. Reconstructing Gas Flows Using Light-Path Approximation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
[22]
Bernhard Kerbl, Georgios Kopanas, Thomas Leimk?hler, and George Drettakis. 2023. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics (ToG) 42, 4 (2023), 1?14.
[23]
Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. 2021. Neural scene flow fields for space-time view synthesis of dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 6498?6508.
[24]
Zhengqi Li, Qianqian Wang, Forrester Cole, Richard Tucker, and Noah Snavely. 2023. Dynibar: Neural dynamic image-based rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 4273?4284.
[25]
David B Lindell, Julien NP Martel, and Gordon Wetzstein. 2021. Autoint: Automatic integration for fast neural volume rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 14556?14565.
[26]
Ricardo Martin-Brualla, Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Dosovitskiy, and Daniel Duckworth. 2021. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 7210?7219.
[27]
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. 2019. Occupancy Networks: Learning 3D Reconstruction in Function Space. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR).
[28]
Marko Mihajlovic, Sergey Prokudin, Marc Pollefeys, and Siyu Tang. 2023. ResFields: Residual Neural Fields for Spatiotemporal Signals. arXiv preprint arXiv:2309.03160 (2023).
[29]
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. 2021. Nerf: Representing scenes as neural radiance fields for view synthesis. Commun. ACM 65, 1 (2021), 99?106.
[30]
Thomas M?ller, Alex Evans, Christoph Schied, and Alexander Keller. 2022. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics (ToG) 41, 4 (2022), 1?15.
[31]
Mohammad Sina Nabizadeh, Stephanie Wang, Ravi Ramamoorthi, and Albert Chern. 2022. Covector Fluids. ACM Transactions on Graphics (TOG) 41, 4 (2022), 113:1?113:15.
[32]
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. 2019. DeepSDF: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 165?174.
[33]
Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, and Ricardo Martin-Brualla. 2021a. Nerfies: Deformable Neural Radiance Fields. In International Conference on Computer Vision (ICCV).
[34]
Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M. Seitz. 2021b. HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields. ACM Trans. Graph. 40, 6, Article 238 (dec 2021).
[35]
Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger. 2020. Convolutional occupancy networks. In Computer Vision?ECCV 2020: 16th European Conference, Glasgow, UK, August 23?28, 2020, Proceedings, Part III 16. Springer, 523?540.
[36]
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. 2020. D-NeRF: Neural Radiance Fields for Dynamic Scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
[37]
Ziyin Qu, Xinxin Zhang, Ming Gao, Chenfanfu Jiang, and Baoquan Chen. 2019. Efficient and Conservative Fluids with Bidirectional Mapping. ACM Transactions on Graphics (Proceedings of SIGGRAPH 2019) 38, 4 (2019).
[38]
Maziar Raissi, Paris Perdikaris, and George E Karniadakis. 2019. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics 378 (2019), 686?707.
[39]
Nicolas Rosset, Guillaume Cordonnier, Regis Duvigneau, and Adrien Bousseau. 2023. Interactive design of 2D car profiles with aerodynamic feedback. In Computer Graphics Forum, Vol. 42. Wiley Online Library, 427?437.
[40]
Francisco Sahli Costabal, Yibo Yang, Paris Perdikaris, Daniel E Hurtado, and Ellen Kuhl. 2020. Physics-informed neural networks for cardiac activation mapping. Frontiers in Physics 8 (2020), 42.
[41]
Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. 2019. PIFu: Pixel-aligned implicit function for high-resolution clothed human digitization. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 2304?2314.
[42]
Takahiro Sato, Christopher Batty, Takeo Igarashi, and Ryoichi Ando. 2018. Spatially adaptive long-term semi-Lagrangian method for accurate velocity advection. Computational Visual Media 4 (2018), 223?230.
[43]
Takahiro Sato, Takeo Igarashi, Christopher Batty, and Ryoichi Ando. 2017. A long-term semi-lagrangian method for accurate velocity advection. In SIGGRAPH Asia 2017 Technical Briefs. 1?4.
[44]
Siyuan Shen, Tianjia Shao, Kun Zhou, Chenfanfu Jiang, Feng Luo, and Yin Yang. 2022. HoD-Net: high-order differentiable deep neural networks and applications. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 36. 8249?8258.
[45]
Vincent Sitzmann, Julien N.P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. 2020. Implicit Neural Representations with Periodic Activation Functions. In Proc. NeurIPS.
[46]
Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. 2020. Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains. NeurIPS (2020).
[47]
Jerry Tessendorf and Brandon Pelfrey. 2011. The characteristic map for fast and efficient vfx fluid simulations. In Computer Graphics International Workshop on VFX, Computer Animation, and Stereo Movies. Ottawa, Canada.
[48]
Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollh?fer, Christoph Lassner, and Christian Theobalt. 2021. Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video. In IEEE International Conference on Computer Vision (ICCV). IEEE.
[49]
Chaoyang Wang, Ben Eckart, Simon Lucey, and Orazio Gallo. 2021a. Neural trajectory fields for dynamic novel view synthesis. arXiv preprint arXiv:2105.05994 (2021).
[50]
Chaoyang Wang, Xueqian Li, Jhony Kaesemodel Pontes, and Simon Lucey. 2022. Neural prior for trajectory estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 6532?6542.
[51]
Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. 2021b. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2106.10689 (2021).
[52]
Qianqian Wang, Yen-Yu Chang, Ruojin Cai, Zhengqi Li, Bharath Hariharan, Aleksander Holynski, and Noah Snavely. 2023. Tracking Everything Everywhere All at Once. In International Conference on Computer Vision.
[53]
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13, 4 (2004), 600?612.
[54]
DC Wiggert and EB Wylie. 1976. Numerical predictions of two-dimensional transient groundwater flow by the method of characteristics. Water Resources Research 12, 5 (1976), 971?977.
[55]
Jinhui Xiong, Ramzi Idoughi, Andres A Aguirre-Pablo, Abdulrahman B Aljedaani, Xiong Dun, Qiang Fu, Sigurdur T Thoroddsen, and Wolfgang Heidrich. 2017. Rainbow particle imaging velocimetry for dense 3D fluid velocity imaging. ACM Transactions on Graphics (TOG) 36, 4 (2017), 1?14.
[56]
Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. 2020. Multiview neural surface reconstruction by disentangling geometry and appearance. Advances in Neural Information Processing Systems 33 (2020), 2492?2502.
[57]
Hong-Xing Yu, Yang Zheng, Yuan Gao, Yitong Deng, Bo Zhu, and Jiajun Wu. 2023. Inferring Hybrid Neural Fluid Fields from Videos. In Thirty-seventh Conference on Neural Information Processing Systems.
[58]
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition. 586?595.
