“Differentiable Dynamic Visible-Light Tomography” by Kang, Bi, Feng, Dong, Zhou, et al. …
Conference:
Type(s):
Title:
- Differentiable Dynamic Visible-Light Tomography
Session/Category Title: \nabla f = ?
Presenter(s)/Author(s):
Abstract:
We propose the first visible-light tomography system for real-time acquisition and reconstruction of general temporally-varying 3D phenomena. Using a single high-speed camera, a high-performance LED array and optical fibers with a total length of 5km, we build a novel acquisition setup with no mechanical movements to simultaneously sample using 1,920 interleaved sources and detectors with a complete 360-degree coverage. Next, we introduce a novel differentiable framework to map both tomography acquisition and reconstruction to a carefully designed autoencoder. This allows the joint and automatic optimization of both processes in an end-to-end fashion, essentially learning to physically compress and computationally decompress the target information. Our framework can adapt to various factors, and trade between capture speed and reconstruction quality. We achieve an acquisition speed of up to 36.8 volumes per second at a spatial resolution of 32x128x128; each volume is captured with as few as 8 images. The effectiveness of the system is demonstrated on acquiring various dynamic scenes. Our results are also validated with the reconstructions computed from the measurements with one source on at a time, and compare favorably with state-of-the-art techniques.
References:
[1]
Anders H Andersen and Avinash C Kak. 1984. Simultaneous algebraic reconstruction technique (SART): a superior implementation of the ART algorithm. Ultrasonic imaging 6, 1 (1984), 81–94.
[2]
Rushil Anirudh, Hyojin Kim, Jayaraman J. Thiagarajan, K. Aditya Mohan, Kyle Champley, and Timo Bremer. 2018. Lose the Views: Limited Angle CT Reconstruction via Implicit Sinogram Completion. In CVPR.
[3]
Y Arai, E Tammisalo, K Iwai, K Hashimoto, and K Shinoda. 1999. Development of a compact computed tomographic apparatus for dental use.Dentomaxillofacial Radiology 28, 4 (1999), 245–248.
[4]
Bradley Atcheson, Ivo Ihrke, Wolfgang Heidrich, Art Tevs, Derek Bradley, Marcus Magnor, and Hans-Peter Seidel. 2008. Time-resolved 3d capture of non-stationary gas flows. TOG 27, 5 (2008), 1–9.
[5]
Mojtaba Bemana, Karol Myszkowski, Jeppe Revall Frisvad, Hans-Peter Seidel, and Tobias Ritschel. 2022. Eikonal fields for refractive novel-view synthesis. In ACM SIGGRAPH 2022 Conference Proceedings. 1–9.
[6]
Guang-Hong Chen, Jie Tang, and Shuai Leng. 2008b. Prior image constrained compressed sensing (PICCS): a method to accurately reconstruct dynamic CT images from highly undersampled projection data sets. Medical physics 35, 2 (2008), 660–663.
[7]
Guang-Hong Chen, Pascal Thériault-Lauzier, Jie Tang, Brian Nett, Shuai Leng, Joseph Zambelli, Zhihua Qi, Nicholas Bevins, Amish Raval, Scott Reeder, 2011. Time-resolved interventional cardiac C-arm cone-beam CT: An application of the PICCS algorithm. IEEE transactions on medical imaging 31, 4 (2011), 907–923.
[8]
Lingyun Chen, Chris C Shaw, Mustafa C Altunbas, Chao-Jen Lai, and Xinming Liu. 2008a. Spatial resolution properties in cone beam CT: a simulation study. Medical physics 35, 2 (2008), 724–734.
[9]
Yi Chen, Yan Xi, and Jun Zhao. 2014. A stationary computed tomography system with cylindrically distributed sources and detectors.Journal of X-ray science and technology 22 6 (2014), 707–25.
[10]
Bruno De Man, Jorge Uribe, Jongduk Baek, Dan Harrison, Zhye Yin, Randy Longtin, Jaydeep Roy, Bill Waters, Colin Wilson, Jonathan Short, 2016. Multisource inverse-geometry CT. Part I. System concept and development. Medical physics 43, 8Part1 (2016), 4607–4616.
[11]
Guanglei Ding, Yitong Liu, Rui Zhang, and Huolin L Xin. 2019. A joint deep learning model to recover information and reduce artifacts in missing-wedge sinograms for electron tomography and beyond. Scientific reports 9, 1 (2019), 1–13.
[12]
M-L Eckert, Wolfgang Heidrich, and Nils Thuerey. 2018. Coupled fluid density and motion from single views. In CGF, Vol. 37. 47–58.
[13]
L. A. Feldkamp, L. C. Davis, and J. W. Kress. 1984. Practical cone-beam algorithm. J. Opt. Soc. Am. A 1, 6 (Jun 1984), 612–619. https://doi.org/10.1364/JOSAA.1.000612
[14]
Richard Gordon, Robert Bender, and Gabor T Herman. 1970. Algebraic reconstruction techniques (ART) for three-dimensional electron microscopy and X-ray photography. Journal of theoretical Biology 29, 3 (1970), 471–481.
[15]
Ida Häggström, C Ross Schmidtlein, Gabriele Campanella, and Thomas J Fuchs. 2019. DeepPET: A deep encoder–decoder network for directly solving the PET image reconstruction inverse problem. Medical image analysis 54 (2019), 253–262.
[16]
Samuel W Hasinoff and Kiriakos N Kutulakos. 2007. Photo-consistent reconstruction of semitransparent scenes by density-sheet decomposition. TPAMI 29, 5 (2007), 870–885.
[17]
Andreas Hauptmann, Simon Arridge, Felix Lucka, Vivek Muthurangu, and Jennifer A Steeden. 2019. Real-time cardiovascular MR with spatio-temporal artifact suppression using deep learning–proof of concept in congenital heart disease. Magnetic resonance in medicine 81, 2 (2019), 1143–1156.
[18]
Godfrey Hounsfield. 1973. Computerized transverse axial scanning (tomography): Part 1. Description of system. The British journal of radiology 46, 552 (1973), 1016–1022.
[19]
Jiang Hsieh. 2003. Computed tomography: principles, design, artifacts, and recent advances. PM344 (2003).
[20]
Jing Huang, Yunwan Zhang, Jianhua Ma, Dong Zeng, Zhaoying Bian, Shanzhou Niu, Qianjin Feng, Zhengrong Liang, and Wufan Chen. 2013. Iterative image reconstruction for sparse-view CT using normal-dose image induced total variation prior. PloS one 8, 11 (2013), e79709.
[21]
Yixing Huang, Xiaolin Huang, Oliver Taubmann, Yan Xia, Viktor Haase, Joachim Hornegger, Guenter Lauritsch, and Andreas Maier. 2017. Restoration of missing data in limited angle tomography based on Helgason–Ludwig consistency conditions. Biomedical Physics & Engineering Express 3, 3 (2017), 035015.
[22]
Matthias B Hullin, Martin Fuchs, Ivo Ihrke, Hans-Peter Seidel, and Hendrik PA Lensch. 2008. Fluorescent immersion range scanning. TOG 27, 3 (2008), 87.
[23]
Ivo Ihrke, B Goidluecke, and Marcus Magnor. 2005. Reconstructing the geometry of flowing water. In ICCV, Vol. 2. IEEE, 1055–1060.
[24]
Ivo Ihrke, Kiriakos N Kutulakos, Hendrik PA Lensch, Marcus Magnor, and Wolfgang Heidrich. 2010. Transparent and specular object reconstruction. In CGF, Vol. 29. 2400–2426.
[25]
Ivo Ihrke and Marcus Magnor. 2004. Image-based tomographic reconstruction of flames. In Proceedings of the 2004 ACM SIGGRAPH/Eurographics symposium on Computer animation. 365–373.
[26]
Kyong Hwan Jin, Michael T McCann, Emmanuel Froustey, and Michael Unser. 2017. Deep convolutional neural network for inverse problems in imaging. TIP 26, 9 (2017), 4509–4522.
[27]
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016. Perceptual losses for real-time style transfer and super-resolution. In ECCV. Springer, 694–711.
[28]
Kaizhang Kang, Zimin Chen, Jiaping Wang, Kun Zhou, and Hongzhi Wu. 2018. Efficient Reflectance Capture Using an Autoencoder. TOG 37, 4, Article 127 (2018), 10 pages.
[29]
Kyeong-Hyeon Kim, Dong-Seok Shin, Sang-Won Kang, Seong-Hee Kang, Tae-Ho Kim, Jin-Beom Chung, Tae Suk Suh, and Dong-Su Kim. 2021. Four-dimensional inverse-geometry computed tomography: a preliminary study. Physics in Medicine & Biology 66, 6 (2021), 065028.
[30]
Yueting Luo, Derrek Spronk, Yueh Z Lee, Otto Zhou, and Jianping Lu. 2021. Simulation on system configuration for stationary head CT using linear carbon nanotube x-ray source arrays. Journal of Medical Imaging 8, 5 (2021), 052114–052114.
[31]
K Aditya Mohan, SV Venkatakrishnan, John W Gibbs, Emine Begum Gulsoy, Xianghui Xiao, Marc De Graef, Peter W Voorhees, and Charles A Bouman. 2015. TIMBIR: A method for time-space reconstruction from interlaced views. IEEE Transactions on Computational Imaging 1, 2 (2015), 96–111.
[32]
P Mozzo, C Procacci, A Tacconi, P Tinazzi Martini, and IA Bergamo Andreis. 1998. A new volumetric CT machine for dental imaging based on the cone-beam technique: preliminary results. European radiology 8, 9 (1998), 1558–1564.
[33]
Thiago Pereira, Szymon Rusinkiewicz, and Wojciech Matusik. 2014. Computational light routing: 3d printed optical fibers for sensing and display. TOG 33, 3 (2014), 1–13.
[34]
Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Lo, Justin Johnson, and Georgia Gkioxari. 2020. Accelerating 3D Deep Learning with PyTorch3D. arXiv:2007.08501 (2020).
[35]
Darius Rückert, Yuanhao Wang, Rui Li, Ramzi Idoughi, and Wolfgang Heidrich. 2022. NeAT: Neural Adaptive Tomography. TOG 41, 4 (2022).
[36]
Marcel Santana Santos, Tsang Ing Ren, and Nima Khademi Kalantari. 2020. Single image HDR reconstruction using a CNN with masked features and perceptual loss. TOG 39, 4 (2020), 80–1.
[37]
Taly Gilat Schmidt, Rebecca Fahrig, Norbert J Pelc, and Edward G Solomon. 2004. An inverse-geometry volumetric CT system with a large-area scanned source: A feasibility study. Medical physics 31, 9 (2004), 2623–2627.
[38]
PR Schwoebel, John M Boone, and Joe Shao. 2014. Studies of a prototype linear stationary x-ray source for tomosynthesis imaging. Physics in Medicine & Biology 59, 10 (2014), 2393.
[39]
Jan-Jakob Sonke, Lambert Zijp, Peter Remeijer, and Marcel Van Herk. 2005. Respiratory correlated cone beam CT. Medical physics 32, 4 (2005), 1176–1186.
[40]
Derrek W Spronk. 2021. Development and Evaluation of a Stationary Head Computed Tomography Scanner. Ph. D. Dissertation. UNC Chapel Hill.
[41]
Nils Thuerey and Tobias Pfaff. 2018. MantaFlow. http://mantaflow.com.
[42]
Borislav Trifonov, Derek Bradley, and Wolfgang Heidrich. 2006. Tomographic Reconstruction of Transparent Objects. In EGSR. 51–60.
[43]
Bo Wang and Huafeng Liu. 2020. FBP-Net for direct reconstruction of dynamic PET images. Physics in Medicine & Biology 65, 23 (2020), 235008.
[44]
Zexiang Xu, Kalyan Sunkavalli, Sunil Hadap, and Ravi Ramamoorthi. 2018. Deep image-based relighting from optimal sparse samples. TOG 37, 4 (2018), 1–13.
[45]
Yidi Yao, Liang Li, and Zhiqiang Chen. 2021. A Novel Static CT System: The Design of Triple Planes CT and its Multi-Energy Simulation Results. Frontiers in Physics (2021), 213.
[46]
Guangming Zang, Ramzi Idoughi, Rui Li, Peter Wonka, and Wolfgang Heidrich. 2021. IntraTomo: self-supervised learning-based tomography via sinogram synthesis and prediction. In ICCV. 1960–1970.
[47]
Guangming Zang, Ramzi Idoughi, Ran Tao, Gilles Lubineau, Peter Wonka, and Wolfgang Heidrich. 2018. Space-Time Tomography for Continuously Deforming Objects. TOG 37, 4, Article 100 (Jul 2018), 14 pages.
[48]
Guangming Zang, Ramzi Idoughi, Ran Tao, Gilles Lubineau, Peter Wonka, and Wolfgang Heidrich. 2019. Warp-and-project tomography for rapidly deforming objects. TOG 38, 4 (2019), 1–13.
[49]
Guangming Zang, Ramzi Idoughi, Congli Wang, Anthony Bennett, Jianguo Du, Scott Skeen, William L Roberts, Peter Wonka, and Wolfgang Heidrich. 2020. TomoFluid: reconstructing dynamic fluid from sparse view videos. In CVPR. 1870–1879.
[50]
J Zhang, R Peng, S Chang, JP Lu, and O Zhou. 2010. Imaging quality assessment of multiplexing x-ray radiography based on multi-beam x-ray source technology. In Medical Imaging 2010: Physics of Medical Imaging, Vol. 7622. SPIE, 1450–1457.
[51]
J Zhang, G Yang, S Chang, JP Lu, and O Zhou. 2008. Hadamard multiplexing radiography based on carbon nanotube field emission multi-pixel x-ray technology. In Medical Imaging 2008: Physics of Medical Imaging, Vol. 6913. SPIE, 628–635.
[52]
Tao Zhang, Hewei Gao, Li Zhang, Yuxiang Xing, and Zhiqiang Chen. 2021. 3D image reconstruction for symmetric-geometry CT with linearly distributed source and detector in a stationary configuration. In Medical Imaging 2021: Physics of Medical Imaging, Vol. 11595. SPIE, 977–980.
[53]
Tao Zhang, Yuxiang Xing, Li Zhang, Xin Jin, Hewei Gao, and Zhiqiang Chen. 2020. Stationary computed tomography with source and detector in linear symmetric geometry: Direct filtered backprojection reconstruction. Medical physics 47, 5 (2020), 2222–2236.
[54]
Shuyao Zhou, Tianqian Zhu, Kanle Shi, Yazi Li, Wen Zheng, and Junhai Yong. 2021. Review of light field technologies. Visual Computing for Industry, Biomedicine, and Art 4, 1 (2021), 29.
