“Colorblind-shareable videos by synthesizing temporal-coherent polynomial coefficients” by Hu, Xueting, Zhuming, Xia, Li, et al. …
Conference:
Type(s):
Title:
- Colorblind-shareable videos by synthesizing temporal-coherent polynomial coefficients
Session/Category Title: Learning from Video
Presenter(s)/Author(s):
Moderator(s):
Abstract:
To share the same visual content between color vision deficiencies (CVD) and normal-vision people, attempts have been made to allocate the two visual experiences of a binocular display (wearing and not wearing glasses) to CVD and normal-vision audiences. However, existing approaches only work for still images. Although state-of-the-art temporal filtering techniques can be applied to smooth the per-frame generated content, they may fail to maintain the multiple binocular constraints needed in our applications, and even worse, sometimes introduce color inconsistency (same color regions map to different colors). In this paper, we propose to train a neural network to predict the temporal coherent polynomial coefficients in the domain of global color decomposition. This indirect formulation solves the color inconsistency problem. Our key challenge is to design a neural network to predict the temporal coherent coefficients, while maintaining all required binocular constraints. Our method is evaluated on various videos and all metrics confirm that it outperforms all existing solutions.
References:
1. 2015. Spectral Edge for displays. https://www.spectraledge.co.uk/spectral-edge-for-displays7. Online; accessed 26-April-2018.Google Scholar
2. Tunç Ozan Aydin, Nikolce Stefanoski, Simone Croci, Markus Gross, and Aljoscha Smolic. 2014. Temporally Coherent Local Tone Mapping of HDR Video. ACM Trans. Graph. 33, 6, Article 196 (Nov. 2014), 13 pages. Google ScholarDigital Library
3. Nicolas Bonneel, Kalyan Sunkavalli, Sylvain Paris, and Hanspeter Pfister. 2013. Example-based Video Color Grading. ACM Trans. Graph. 32, 4, Article 39 (July 2013), 12 pages. Google ScholarDigital Library
4. Nicolas Bonneel, Kalyan Sunkavalli, James Tompkin, Deqing Sun, Sylvain Paris, and Hanspeter Pfister. 2014. Interactive Intrinsic Video Editing. ACM Trans. Graph. 33, 6, Article 197 (Nov. 2014), 10 pages. Google ScholarDigital Library
5. Nicolas Bonneel, James Tompkin, Deqing Sun, Oliver Wang, Kalyan Sunkavalli, Sylvain Paris, and Hanspeter Pfister. 2017. Consistent video filtering for camera arrays. In Computer Graphics Forum, Vol. 36. Wiley Online Library, 397–407.Google Scholar
6. Nicolas Bonneel, James Tompkin, Kalyan Sunkavalli, Deqing Sun, Sylvain Paris, and Hanspeter Pfister. 2015. Blind Video Temporal Consistency. ACM Trans. Graph. 34, 6, Article 196 (Oct. 2015), 9 pages. Google ScholarDigital Library
7. Dongdong Chen, Jing Liao, Lu Yuan, Nenghai Yu, and Gang Hua. 2017. Coherent online video style transfer. In Proc. Intl. Conf. Computer Vision (ICCV).Google ScholarCross Ref
8. Soon Hau Chua, Haimo Zhang, Muhammad Hammad, Shengdong Zhao, Sahil Goyal, and Karan Singh. 2015. ColorBless: Augmenting Visual Information for Colorblind People with Binocular Luster Effect. ACM Trans. Comput.-Hum. Interact. 21, 6, Article 32 (Jan. 2015), 20 pages. Google ScholarDigital Library
9. Eugenio Culurciello and Alfredo Canziani. 2017. e-Lab Video Data Set. https://engineering.purdue.edu/elab/eVDS/.Google Scholar
10. Qingnan Fan, Jiaolong Yang, David P. Wipf, Baoquan Chen, and Xin Tong. 2018. Image Smoothing via Unsupervised Learning. CoRR abs/1811.02804 (2018). arXiv:1811.02804 http://arxiv.org/abs/1811.02804Google Scholar
11. Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics. 249–256.Google Scholar
12. Xinghong Hu, Zhuming Zhang, Xueting Liu, and Tien-Tsin Wong. 2019. Deep Visual Sharing with Colorblind. IEEE Transactions on Computational Imaging (2019).Google ScholarCross Ref
13. Yuanming Hu, Hao He, Chenxi Xu, Baoyuan Wang, and Stephen Lin. 2018. Exposure: A White-Box Photo Post-Processing Framework. ACM Transactions on Graphics (TOG) 37, 2 (2018), 26.Google ScholarDigital Library
14. Chun-Rong Huang, Kuo-Chuan Chiu, and Chu-Song Chen. 2011. Temporal color consistency-based video reproduction for dichromats. IEEE Transactions on Multimedia 13, 5 (2011), 950–960.Google ScholarDigital Library
15. Jia-Bin Huang, Yu-Cheng Tseng, Se-In Wu, and Sheng-Jyh Wang. 2007. Information preserving color transformation for protanopia and deuteranopia. Signal Processing Letters, IEEE 14, 10 (2007), 711–714.Google ScholarCross Ref
16. Mark J Huiskes and Michael S Lew. 2008. The MIR flickr retrieval evaluation. In Proceedings of the 1st ACM international conference on Multimedia information retrieval. ACM, 39–43.Google ScholarDigital Library
17. Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox. 2017. FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 1647–1655.Google ScholarCross Ref
18. Bernd Jähne, Horst Haussecker, and Peter Geissler. 1999. Handbook of computer vision and applications. Vol. 2. Citeseer.Google Scholar
19. Huaizu Jiang, Deqing Sun, Varun Jampani, Ming-Hsuan Yang, Erik Learned-Miller, and Jan Kautz. 2017. Super slomo: High quality estimation of multiple intermediate frames for video interpolation. arXiv preprint arXiv:1712.00080 (2017).Google Scholar
20. Tony Johnson. 1996. Methods for characterizing colour scanners and digital cameras. Displays 16, 4 (1996), 183–191.Google ScholarCross Ref
21. D Kinga and J Ba Adam. 2015. A method for stochastic optimization. In International Conference on Learning Representations (ICLR), Vol. 5.Google Scholar
22. Naejin Kong, Peter V. Gehler, and Michael J. Black. 2014. Intrinsic Video. In Computer Vision – ECCV 2014 (Lecture Notes in Computer Science), Vol. 8690. Springer International Publishing, 360–375.Google Scholar
23. Wei-Sheng Lai, Jia-Bin Huang, Oliver Wang, Eli Shechtman, Ersin Yumer, and Ming-Hsuan Yang. 2018. Learning blind video temporal consistency. In Proceedings of the European Conference on Computer Vision (ECCV). 170–185.Google ScholarCross Ref
24. Manuel Lang, Oliver Wang, Tunc Aydin, Aljoscha Smolic, and Markus Gross. 2012. Practical Temporal Consistency for Image-based Graphics Applications. ACM Trans. Graph. 31, 4, Article 34 (July 2012), 8 pages. Google ScholarDigital Library
25. Cewu Lu, Li Xu, and Jiaya Jia. 2012. Contrast preserving decolorization. In Computational Photography (ICCP), 2012 IEEE International Conference on. IEEE, 1–7.Google Scholar
26. Gustavo M Machado and Manuel M Oliveira. 2010. Real-Time Temporal-Coherent Color Contrast Enhancement for Dichromats. In Computer Graphics Forum, Vol. 29. Wiley Online Library, 933–942.Google Scholar
27. Gustavo M Machado, Manuel M Oliveira, and Leandro AF Fernandes. 2009. A physiologically-based model for simulation of color vision deficiency. IEEE Transactions on Visualization and Computer Graphics 15, 6 (2009), 1291–1298.Google ScholarDigital Library
28. Sylvain Paris. 2008. Edge-Preserving Smoothing and Mean-Shift Segmentation of Video Streams. Springer Berlin Heidelberg, Berlin, Heidelberg, 460–473. Google ScholarDigital Library
29. Behzad Sajadi, Aditi Majumder, Manuel M Oliveira, Rosália G Schneider, and Ramesh Raskar. 2013. Using patterns to encode color information for dichromats. IEEE transactions on visualization and computer graphics 19, 1 (2013), 118–129.Google ScholarDigital Library
30. Mehdi SM Sajjadi, Raviteja Vemulapalli, and Matthew Brown. 2018. Frame-Recurrent Video Super-Resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6626–6634.Google ScholarCross Ref
31. Wuyao Shen, Xiangyu Mao, Xinghong Hu, and Tien-Tsin Wong. 2016. Seamless Visual Sharing with Color Vision Deficiencies. ACM Trans. Graph. 35, 4, Article 70 (July 2016), 12 pages. Google ScholarDigital Library
32. Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. 2012. UCF101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402 (2012).Google Scholar
33. Chung-Ming Wang, Yao-Hsien Huang, and Ming-Long Huang. 2006. An effective algorithm for image sequence color transfer. Mathematical and Computer Modelling 44, 7–8 (2006), 608–627. Google ScholarDigital Library
34. Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. 2018. Video-to-video synthesis. arXiv preprint arXiv:1808.06601 (2018).Google ScholarDigital Library
35. Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13, 4 (2004), 600–612.Google ScholarDigital Library
36. Stephen Wolf. 2003. Color correction matrix for digital still and video imaging systems. National Telecommunications and Information Administration Washington, DC.Google Scholar
37. Xuan Yang, Linling Zhang, Tien-Tsin Wong, and Pheng-Ann Heng. 2012. Binocular Tone Mapping. ACM Transactions on Graphics 31, 4 (2012), 93:1–93:10.Google ScholarDigital Library
38. Genzhi Ye, Elena Garces, Yebin Liu, Qionghai Dai, and Diego Gutierrez. 2014. Intrinsic Video and Applications. ACM Trans. Graph. 33, 4, Article 80 (July 2014), 11 pages. Google ScholarDigital Library
39. Zhuming Zhang, Chu Han, Shengfeng He, Xueting Liu, Haichao Zhu, Xinghong Hu, and Tien-Tsin Wong. 2019. Deep binocular tone mapping. The Visual Computer (2019), 1–15. https://link.springer.com/article/10.1007/s00371-019-01669-8Google Scholar
40. Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. 2018. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence 40, 6 (2018), 1452–1464.Google Scholar
