“Image smoothing via unsupervised learning”
Conference:
Type(s):
Title:
- Image smoothing via unsupervised learning
Session/Category Title: Low-level imaging
Presenter(s)/Author(s):
Moderator(s):
Abstract:
Image smoothing represents a fundamental component of many disparate computer vision and graphics applications. In this paper, we present a unified unsupervised (label-free) learning framework that facilitates generating flexible and high-quality smoothing effects by directly learning from data using deep convolutional neural networks (CNNs). The heart of the design is the training signal as a novel energy function that includes an edge-preserving regularizer which helps maintain important yet potentially vulnerable image structures, and a spatially-adaptive Lp flattening criterion which imposes different forms of regularization onto different image regions for better smoothing quality. We implement a diverse set of image smoothing solutions employing the unified framework targeting various applications such as, image abstraction, pencil sketching, detail enhancement, texture removal and content-aware image manipulation, and obtain results comparable with or better than previous methods. Moreover, our method is extremely fast with a modern GPU (e.g, 200 fps for 1280×720 images).
References:
1. Francis Bach, Rodolphe Jenatton, Julien Mairal, Guillaume Obozinski, et al. 2012. Structured sparsity through convex optimization. Statist. Sci. 27, 4 (2012), 450–468.Google ScholarCross Ref
2. Linchao Bao, Yibing Song, Qingxiong Yang, Hao Yuan, and Gang Wang. 2014. Tree filtering: Efficient structure-preserving smoothing with a minimum spanning tree. IEEE Transactions on Image Processing 23, 2 (2014), 555–569. Google ScholarDigital Library
3. S. Bi, X. Han, and Y. Yu. 2015. An L1 image transform for edge-preserving smoothing and scene-level intrinsic decomposition. ACM Transactions on Graphics 34, 4 (2015), 78. Google ScholarDigital Library
4. Dongdong Chen, Jing Liao, Lu Yuan, Nenghai Yu, and Gang Hua. 2017. Coherent online video style transfer. In Proc. Intl. Conf. Computer Vision (ICCV).Google ScholarCross Ref
5. Dongdong Chen, Lu Yuan, Jing Liao, Nenghai Yu, and Gang Hua. 2017b. Stylebank: An explicit representation for neural image style transfer. In Proc. CVPR, Vol. 1. 4.Google Scholar
6. Dongdong Chen, Lu Yuan, Jing Liao, Nenghai Yu, and Gang Hua. 2018. Stereoscopic neural style transfer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Vol. 10.Google ScholarCross Ref
7. Jiawen Chen, Sylvain Paris, and Frédo Durand. 2007. Real-time edge-aware image processing with the bilateral grid. ACM Transactions on Graphics 26, 3 (2007), 103. Google ScholarDigital Library
8. Qifeng Chen, Jia Xu, and Vladlen Koltun. 2017a. Fast image processing with fully-convolutional networks. In International Conference on Computer Vision. 2497–2506.Google ScholarCross Ref
9. Hojin Cho, Hyunjoon Lee, Henry Kang, and Seungyong Lee. 2014. Bilateral texture filtering. ACM Transactions on Graphics 33, 4 (2014), 128. Google ScholarDigital Library
10. Ginmo Chung and Luminita A Vese. 2009. Image segmentation using a multilayer level-set approach. Computing and Visualization in Science 12, 6 (2009), 267–285.Google ScholarDigital Library
11. Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. 2010. The PASCAL visual object classes (VOC) challenge. International Journal of Computer Vision 88, 2 (2010), 303–338. Google ScholarDigital Library
12. Qingnan Fan, Jiaolong Yang, Gang Hua, Baoquan Chen, and David Wipf. 2017. A generic deep architecture for single image reflection removal and image smoothing. In International Conference on Computer Vision. 3238–3247.Google ScholarCross Ref
13. Qingnan Fan, Jiaolong Yang, Gang Hua, Baoquan Chen, and David Wipf. 2018. Revisiting Deep Intrinsic Image Decompositions. In Proceedings of the IEEE conference on computer vision and pattern recognition.Google ScholarCross Ref
14. Qingnan Fan, Fan Zhong, Dani Lischinski, Daniel Cohen-Or, and Baoquan Chen. 2015. JumpCut: non-successive mask transfer and interpolation for video cutout. ACM Transactions on Graphics 34, 6 (2015), 195. Google ScholarDigital Library
15. Zeev Farbman, Raanan Fattal, Dani Lischinski, and Richard Szeliski. 2008. Edge-preserving decompositions for multi-scale tone and detail manipulation. ACM Transactions on Graphics 27, 3 (2008), 67. Google ScholarDigital Library
16. Raanan Fattal. 2009. Edge-avoiding wavelets and their applications. ACM Transactions on Graphics 28, 3 (2009), 22. Google ScholarDigital Library
17. Eduardo SL Gastal and Manuel M Oliveira. 2011. Domain transform for edge-aware image and video processing. In ACM Transactions on Graphics, Vol. 30. ACM, 69. Google ScholarDigital Library
18. Michaël Gharbi, Jiawen Chen, Jonathan T Barron, Samuel W Hasinoff, and Frédo Durand. 2017. Deep bilateral learning for real-time image enhancement. ACM Transactions on Graphics (TOG) 36, 4 (2017), 118. Google ScholarDigital Library
19. Bumsub Ham, Minsu Cho, and Jean Ponce. 2015. Robust image filtering using joint static and dynamic guidance. In IEEE Conference on Computer Vision and Pattern Recognition. 4823–4831.Google ScholarCross Ref
20. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In IEEE International Conference on Computer Vision. 1026–1034. Google ScholarDigital Library
21. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition. 770–778.Google ScholarCross Ref
22. Mingming He, Dongdong Chen, Jing Liao, Pedro V Sander, and Lu Yuan. 2018. Deep Exemplar-based Colorization. ACM Transactions on Graphics (Proc. of Siggraph 2018) (2018). Google ScholarDigital Library
23. Paul W Holland and Roy E Welsch. 1977. Robust regression using iteratively reweighted least-squares. Communications in Statistics-theory and Methods 6, 9 (1977), 813–827.Google ScholarCross Ref
24. Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning. 448–456. Google ScholarDigital Library
25. Michael Kass and Justin Solomon. 2010. Smoothed local histogram filters. In ACM Transactions on Graphics, Vol. 29. 100. Google ScholarDigital Library
26. Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. 2016. Accurate image super-resolution using very deep convolutional networks. In IEEE Conference on Computer Vision and Pattern Recognition. 1646–1654.Google ScholarCross Ref
27. Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. International Conference on Learning Representations (2015).Google Scholar
28. Sifei Liu, Jinshan Pan, and Ming-Hsuan Yang. 2016. Learning recursive filters for low-level vision via a hybrid neural network. In European Conference on Computer Vision. 560–576.Google ScholarCross Ref
29. Wei Liu, Xiaogang Chen, Chuanhua Shen, Zhi Liu, and Jie Yang. 2017. Semi-Global Weighted Least Squares in Image Filtering. In IEEE International Conference on Computer Vision.Google Scholar
30. Cewu Lu, Li Xu, and Jiaya Jia. 2012. Combining sketch and tone for pencil drawing production. In Proceedings of the Symposium on Non-Photorealistic Animation and Rendering. Eurographics Association, 65–73. Google ScholarDigital Library
31. Kaiyue Lu, Shaodi You, and Nick Barnes. 2018. Deep Texture and Structure Aware Filtering Network for Image Smoothing. EuropeanConferenceonComputerVision(ECCV)Google Scholar
32. Dongbo Min, Sunghwan Choi, Jiangbo Lu, Bumsub Ham, Kwanghoon Sohn, and Minh N Do. 2014. Fast global image smoothing based on weighted least squares. IEEE Transactions on Image Processing 23, 12 (2014), 5638–5653.Google ScholarCross Ref
33. Pavel Mrázek, Joachim Weickert, and Andres Bruhn. 2006. On robust estimation and smoothing with spatial and tonal kernels. In Geometric properties for incomplete data. Springer, 335–352.Google Scholar
34. Sylvain Paris and Frédo Durand. 2006. A fast approximation of the bilateral filter using a signal processing approach. In European Conference on Computer Vision. 568–580. Google ScholarDigital Library
35. Sylvain Paris, Samuel W Hasinoff, and Jan Kautz. 2011. Local Laplacian filters: Edge-aware image processing with a Laplacian pyramid. ACM Trans. Graph. 30, 4 (2011), 68–1. Google ScholarDigital Library
36. Pietro Perona and Jitendra Malik. 1990. Scale-space and edge detection using anisotropic diffusion. IEEE Transactions on Pattern Analysis and Machine Intelligence 12, 7 (1990), 629–639. Google ScholarDigital Library
37. VB Surya Prasath, Dmitry Vorotnikov, Rengarajan Pelapur, Shani Jose, Guna Seetharaman, and Kannappan Palaniappan. 2015. Multiscale Tikhonov-total variation image restoration using spatially varying edge coherence exponent. IEEE Transactions on Image Processing 24, 12 (2015), 5220–5235.Google ScholarDigital Library
38. Yaniv Romano, Michael Elad, and Peyman Milanfar. 2017. The little engine that could: Regularization by denoising (RED). SIAM Journal on Imaging Sciences 10, 4 (2017), 1804–1844.Google ScholarCross Ref
39. Guodong Rong and Tiow-Seng Tan. 2006. Jump flooding in GPU with applications to Voronoi diagram and distance transform. In The 2006 symposium on Interactive 3D graphics and games. 109–116. Google ScholarDigital Library
40. Carlo Tomasi. 1998. Bilateral filtering for gray and color images. In International Conference on Computer Vision. IEEE, 839–846. Google ScholarDigital Library
41. Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. 2018. Deep Image Prior. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018).Google Scholar
42. Ben Weiss. 2006. Fast median and bilateral filtering. Acm Transactions on Graphics 25, 3 (2006), 519–526. Google ScholarDigital Library
43. Holger Winnemöller, Sven C Olsen, and Bruce Gooch. 2006. Real-time video abstraction. In ACM Transactions On Graphics, Vol. 25. 1221–1226. Google ScholarDigital Library
44. Li Xu, Cewu Lu, Yi Xu, and Jiaya Jia. 2011. Image smoothing via L0 gradient minimization. In ACM Transactions on Graphics, Vol. 30. 174. Google ScholarDigital Library
45. Li Xu, Jimmy SJ. Ren, Qiong Yan, Renjie Liao, and Jiaya Jia. 2015. Deep edge-aware filters. In International Conference on Machine Learning. 1669–1678. Google ScholarDigital Library
46. Li Xu, Qiong Yan, Yang Xia, and Jiaya Jia. 2012. Structure extraction from texture via relative total variation. ACM Transactions on Graphics 31, 6 (2012), 139. Google ScholarDigital Library
47. Fisher Yu and Vladlen Koltun. 2016. Multi-scale context aggregation by dilated convolutions. International Conference on Learning Representations.Google Scholar
48. Feihu Zhang, Longquan Dai, Shiming Xiang, and Xiaopeng Zhang. 2015. Segment graph based image filtering: fast structure-preserving smoothing. In IEEE International Conference on Computer Vision. 361–369. Google ScholarDigital Library
49. Pingping Zhang, Dong Wang, Huchuan Lu, Hongyu Wang, and Xiang Ruan. 2017. Amulet: Aggregating multi-level convolutional features for salient object detection. In International Conference on Computer Vision. 202–211.Google ScholarCross Ref
50. Qi Zhang, Xiaoyong Shen, Li Xu, and Jiaya Jia. 2014. Rolling Guidance Filter. In European Conference on Computer Vision. 815–830.Google Scholar
51. Feiyun Zhu, Ying Wang, Bin Fan, Shiming Xiang, Geofeng Meng, and Chunhong Pan. 2014. Spectral unmixing via data-guided sparsity. IEEE Transactions on Image Processing 23, 12 (2014), 5412–5427.Google ScholarCross Ref
