“Mob-FGSR: Frame Generation and Super Resolution for Mobile Real-time Rendering”
Conference:
Type(s):
Title:
- Mob-FGSR: Frame Generation and Super Resolution for Mobile Real-time Rendering
Presenter(s)/Author(s):
Abstract:
We propose Mob-FGSR, a lightweight supersampling framework designed specifically for mobile devices, integrating frame generation with super-resolution to significantly enhance real-time rendering performance. This method operates efficiently without high-end GPUs or dedicated hardware, surpassing other lightweight solutions in extensive testing, and shows great potential for mobile real-time rendering applications.
References:
[1]
Kurt Akeley. 1993. Reality engine graphics. In Proceedings of the 20th Annual Conference on Computer Graphics and Interactive Techniques. 109?116.
[2]
Tomas Akenine-Mo, Eric Haines, Naty Hoffman, 2018. Real-time rendering. (2018).
[3]
AMD. 2022. FidelityFX Super Resolution 2.0. https://gpuopen.com/fidelityfx-superresolution-2/.
[4]
AMD. 2023. AMD FSR 3 Now Available. https://community.amd.com/t5/gaming/amd-fsr-3-now-available/ba-p/634265.
[5]
Dmitry Andreev. 2010. Real-time frame rate up-conversion for video games: or how to get from 30 to 60 fps for” free”. In ACM SIGGRAPH 2010 Talks. 1?1.
[6]
Connelly Barnes, Eli Shechtman, Dan B Goldman, and Adam Finkelstein. 2010. The generalized patchmatch correspondence algorithm. In Computer Vision?ECCV 2010: 11th European Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part III 11. Springer, 29?43.
[7]
Huw Bowles, Kenny Mitchell, Robert W Sumner, Jeremy Moore, and Markus Gross. 2012. Iterative image warping. In Computer graphics forum, Vol. 31. Wiley Online Library, 237?246.
[8]
Karlis Martins Briedis, Abdelaziz Djelouah, Mark Meyer, Ian McGonigal, Markus Gross, and Christopher Schroers. 2021. Neural frame interpolation for rendered content. ACM Transactions on Graphics (TOG) 40, 6 (2021), 1?13.
[9]
Karlis Martins Briedis, Abdelaziz Djelouah, Rapha?l Ortiz, Mark Meyer, Markus Gross, and Christopher Schroers. 2023. Kernel-Based Frame Interpolation for Spatio-Temporally Adaptive Rendering. In ACM SIGGRAPH 2023 Conference Proceedings. 1?11.
[10]
Robert L Cook. 1986. Stochastic sampling in computer graphics. ACM Transactions on Graphics (TOG) 5, 1 (1986), 51?72.
[11]
Andrew Edelsten, Paula Jukarainen, and Anjul Patney. 2019. Truly next-gen: Adding deep learning to games and graphics. In Game Developers Conference.
[12]
Epic. 2022. Temporal Super Resolution. https://docs.unrealengine.com/5.2/en-US/temporal-super-resolution-in-unreal-engine/.
[13]
Epic Games. 2023. The most powerful real-time 3D creation tool – Unreal Engine. https://www.unrealengine.com/en-US
[14]
Jie Guo, Xihao Fu, Liqiang Lin, Hengjun Ma, Yanwen Guo, Shiqiu Liu, and Ling-Qi Yan. 2021. Extranet: Real-time extrapolated rendering for low-latency temporal supersampling. ACM Transactions on Graphics (TOG) 40, 6 (2021), 1?16.
[15]
Johannes Hanika, Lorenzo Tessari, and Carsten Dachsbacher. 2021. Fast temporal reprojection without motion vectors. Journal of Computer Graphics Techniques Vol 10, 3 (2021).
[16]
Sam Holmes and Jonathan Wicks. 2020. QCOM Frame Extrapolation. https://registry.khronos.org/OpenGL/extensions/QCOM/QCOM_frame_extrapolation.txt
[17]
Yinlin Hu, Rui Song, and Yunsong Li. 2016. Efficient coarse-to-fine patchmatch for large displacement optical flow. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5704?5712.
[18]
Jorge Jimenez, Jose I Echevarria, Tiago Sousa, and Diego Gutierrez. 2012. SMAA: Enhanced subpixel morphological antialiasing. Computer Graphics Forum 31, 2pt1 (2012), 355?364.
[19]
Khronos. 2022. OpenGL 4 Reference Pages – imageAtomicMin. https://registry.khronos.org/OpenGL-Refpages/gl4/html/imageAtomicMin.xhtml.
[20]
Sungkil Lee, Younguk Kim, and Elmar Eisemann. 2018. Iterative depth warping. ACM Transactions on Graphics (TOG) 37, 5 (2018), 1?13.
[21]
Zhan Li, Carl S Marshall, Deepak S Vembar, and Feng Liu. 2022. Future Frame Synthesis for Fast Monte Carlo Rendering. In Graphics Interface.
[22]
Edward Liu. 2020. DLSS 2.0-Image Reconstruction for Real-time Rendering with Deep Learning. In GPU Technology Conference (GTC).
[23]
William R Mark, Leonard McMillan, and Gary Bishop. 1997. Post-rendering 3D warping. In Proceedings of the 1997 Symposium on Interactive 3D Graphics. 7?16.
[24]
NVIDIA. 2023. DLSS 3: AI-Powered Neural Graphics Innovations. https://www.nvidia.com/en-sg/geforce/news/dlss3-ai-powered-neural-graphics-innovations/.
[25]
Qualcomm. 2023. Mobile Gaming. https://developer.qualcomm.com/solutions/mobile-gaming
[26]
Alexander Reshetov. 2009. Morphological antialiasing. In Proceedings of the Conference on High Performance Graphics 2009. 109?116.
[27]
Iain E Richardson. 2004. H. 264 and MPEG-4 video compression: video coding for next-generation multimedia. John Wiley & Sons.
[28]
Andre Schollmeyer, Simon Schneegans, Stephan Beck, Anthony Steed, and Bernd Froehlich. 2017. Efficient hybrid image warping for high frame-rate stereoscopic rendering. IEEE Transactions on Visualization and Computer Graphics 23, 4 (2017), 1332?1341.
[29]
Jonathan Shade, Steven Gortler, Li-wei He, and Richard Szeliski. 1998. Layered depth images. In Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques. 231?242.
[30]
Manu Mathew Thomas, Karthik Vaidyanathan, Gabor Liktor, and Angus G Forbes. 2020. A reduced-precision network for image reconstruction. ACM Transactions on Graphics (TOG) 39, 6 (2020), 1?12.
[31]
Unity. 2023. Unity Engine. https://unity.com/products/unity-engine
[32]
Songyin Wu, Sungye Kim, Zheng Zeng, Deepak Vembar, Sangeeta Jha, Anton Kaplanyan, and Ling-Qi Yan. 2023a. ExtraSS: A Framework for Joint Spatial Super Sampling and Frame Extrapolation. In Proceedings of the 2023 SIGGRAPH Asia Conference. 1?11.
[33]
Zhizhen Wu, Chenyu Zuo, Yuchi Huo, Yazhen Yuan, Yifan Peng, Guiyang Pu, Rui Wang, and Hujun Bao. 2023b. Adaptive recurrent frame prediction with learnable motion vectors. In SIGGRAPH Asia 2023 Conference Papers. 1?11.
[34]
Lei Xiao, Salah Nouri, Matt Chapman, Alexander Fix, Douglas Lanman, and Anton Kaplanyan. 2020. Neural supersampling for real-time rendering. ACM Transactions on Graphics (TOG) 39, 4 (2020), 142?1?12.
[35]
Xiangyu Xu, Li Siyao, Wenxiu Sun, Qian Yin, and Ming-Hsuan Yang. 2019. Quadratic video interpolation. Advances in Neural Information Processing Systems 32 (2019).
[36]
Lei Yang, Shiqiu Liu, and Marco Salvi. 2020. A survey of temporal antialiasing techniques. Computer Graphics Forum 39, 2 (2020), 607?621.
[37]
Lei Yang, Yu-Chiu Tse, Pedro V Sander, Jason Lawrence, Diego Nehab, Hugues Hoppe, and Clara L Wilkins. 2011. Image-based bidirectional scene reprojection. In Proceedings of the 2011 SIGGRAPH Asia Conference. 1?10.
[38]
Sipeng Yang, Yunlu Zhao, Yuzhe Luo, He Wang, Hongyu Sun, Chen Li, Binghuang Cai, and Xiaogang Jin. 2023. MNSS: Neural Supersampling Framework for Real-Time Rendering on Mobile Devices. IEEE Transactions on Visualization and Computer Graphics (2023), 1?14.
[39]
Zheng Zeng, Shiqiu Liu, Jinglei Yang, Lu Wang, and Ling-Qi Yan. 2021. Temporally Reliable Motion Vectors for Real-time Ray Tracing. Computer Graphics Forum 40, 2 (2021), 79?90.
[40]
Zhihua Zhong, Jingsen Zhu, Yuxin Dai, Chuankun Zheng, Guanlin Chen, Yuchi Huo, Hujun Bao, and Rui Wang. 2023. FuseSR: Super Resolution for Real-time Rendering through Efficient Multi-resolution Fusion. In SIGGRAPH Asia 2023 Conference Papers. 1?10.
