“Semantic soft segmentation” by Aksoy, Oh, Paris, Pollefeys and Matusik

  • ©Yagiz Aksoy, Tae-Hyun Oh, Sylvain Paris, Marc Pollefeys, and Wojciech Matusik



Entry Number: 72

Session Title:

    Image & Shape Analysis With CNNs


    Semantic soft segmentation




    Accurate representation of soft transitions between image regions is essential for high-quality image editing and compositing. Current techniques for generating such representations depend heavily on interaction by a skilled visual artist, as creating such accurate object selections is a tedious task. In this work, we introduce semantic soft segments, a set of layers that correspond to semantically meaningful regions in an image with accurate soft transitions between different objects. We approach this problem from a spectral segmentation angle and propose a graph structure that embeds texture and color features from the image as well as higher-level semantic information generated by a neural network. The soft segments are generated via eigendecomposition of the carefully constructed Laplacian matrix fully automatically. We demonstrate that otherwise complex image editing tasks can be done with little effort using semantic soft segments.


    1. R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua, and S. Süsstrunk. 2012. SLIC Superpixels Compared to State-of-the-Art Superpixel Methods. IEEE Trans. Pattern Anal. Mach. Intell. 34, 11 (2012), 2274–2282. Google ScholarDigital Library
    2. Yağiz Aksoy, Tunç Ozan Aydin, and Marc Pollefeys. 2017a. Designing Effective Inter-Pixel Information Flow for Natural Image Matting. In Proc. CVPR.Google ScholarCross Ref
    3. Yağiz Aksoy, Tunç Ozan Aydin, Marc Pollefeys, and Aljoša Smolić. 2016. Interactive High-Quality Green-Screen Keying via Color Unmixing. ACM Trans. Graph. 35, 5 (2016), 152:1–152:12. Google ScholarDigital Library
    4. Yağiz Aksoy, Tunç Ozan Aydin, Aljoša Smolić, and Marc Pollefeys. 2017b. Unmixing-Based Soft Color Segmentation for Image Manipulation. ACM Trans. Graph. 36, 2 (2017), 19:1–19:19. Google ScholarDigital Library
    5. Xiaobo An and Fabio Pellacini. 2008. AppProp: All-pairs Appearance-space Edit Propagation. ACM Trans. Graph. 27, 3 (2008), 40:1–40:9. Google ScholarDigital Library
    6. R. Barrett, M. Berry, T. Chan, J. Demmel, J. Donato, J. Dongarra, V. Eijkhout, R. Pozo, C. Romine, and H. van der Vorst. 1994. Templates for the Solution of Linear Systems: Building Blocks for Iterative Methods. SIAM.Google Scholar
    7. Gedas Bertasius, Jianbo Shi, and Lorenzo Torresani. 2015. High-for-low and low-for-high: Efficient boundary detection from deep object features and its applications to high-level vision. In Proc. ICCV. Google ScholarDigital Library
    8. Gedas Bertasius, Jianbo Shi, and Lorenzo Torresani. 2016. Semantic Segmentation with Boundary Neural Fields. In Proc. CVPR.Google ScholarCross Ref
    9. V. Bychkovsky, S. Paris, E. Chan, and F. Durand. 2011. Learning Photographic Global Tonal Adjustment with a Database of Input/Output Image Pairs. In Proc. CVPR. Google ScholarDigital Library
    10. Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. 2016. COCO-Stuff: Thing and Stuff Classes in Context. arXiv: 1612.03716 {cs.CV} (2016).Google Scholar
    11. Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. 2017. DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs. IEEE Trans. Pattern Anal. Mach. Intell. (2017).Google Scholar
    12. Qifeng Chen, Dingzeyu Li, and Chi-Keung Tang. 2013. KNN Matting. IEEE Trans. Pattern Anal. Mach. Intell. 35, 9 (2013), 2175–2188. Google ScholarDigital Library
    13. Xiaowu Chen, Dongqing Zou, Qinping Zhao, and Ping Tan. 2012. Manifold Preserving Edit Propagation. ACM Trans. Graph. 31, 6 (2012), 132:1–132:7. Google ScholarDigital Library
    14. Yuki Endo, Satoshi Iizuka, Yoshihiro Kanamori, and Jun Mitani. 2016. DeepProp: Extracting Deep Features from a Single Image for Edit Propagation. Comput. Graph. Forum 35, 2 (2016), 189–201.Google ScholarCross Ref
    15. D. Eynard, A. Kovnatsky, and M. M. Bronstein. 2014. Laplacian colormaps: a framework for structure-preserving color transformations. Comput. Graph. Forum 33, 2 (2014), 215–224. Google ScholarDigital Library
    16. H. Farid and E. P. Simoncelli. 2004. Differentiation of discrete multidimensional signals. IEEE Trans. Image Process. 13, 4 (2004), 496–508. Google ScholarDigital Library
    17. Alireza Fathi, Zbigniew Wojna, Vivek Rathod, Peng Wang, Hyun Oh Song, Sergio Guadarrama, and Kevin P. Murphy. 2017. Semantic Instance Segmentation via Deep Metric Learning. arXiv: 1703.10277 {cs.CV} (2017).Google Scholar
    18. Bharath Hariharan, Pablo Arbeláez, Ross Girshick, and Jitendra Malik. 2015. Hyper-columns for object segmentation and fine-grained localization. In Proc. CVPR.Google ScholarCross Ref
    19. Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Girshick. 2017. Mask R-CNN. In Proc. ICCV.Google Scholar
    20. Kaiming He, Jian Sun, and Xiaoou Tang. 2013. Guided Image Filtering. IEEE Trans. Pattern Anal. Mach. Intell. 35, 6 (2013), 1397–1409. Google ScholarDigital Library
    21. Elad Hoffer and Nir Ailon. 2015. Deep metric learning using triplet network. In International Workshop on Similarity-Based Pattern Recognition.Google ScholarCross Ref
    22. Anat Levin, Dani Lischinski, and Yair Weiss. 2008a. A Closed-Form Solution to Natural Image Matting. IEEE Trans. Pattern Anal. Mach. Intell. 30, 2 (2008), 228–242. Google ScholarDigital Library
    23. Anat Levin, Alex Rav-Acha, and Dani Lischinski. 2008b. Spectral Matting. IEEE Trans. Pattern Anal. Mach. Intell. 30, 10 (2008), 1699–1712. Google ScholarDigital Library
    24. Y. Li, E. Adelson, and A. Agarwala. 2008. ScribbleBoost: Adding Classification to Edge-Aware Interpolation of Local Image and Video Adjustments. Comput. Graph. Forum 27, 4 (2008), 1255–1264. Google ScholarDigital Library
    25. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft COCO: Common objects in context. In Proc. ECCV.Google ScholarCross Ref
    26. Tae-Hyun Oh, Kyungdon Joo, Neel Joshi, Baoyuan Wang, In So Kweon, and Sing Bing Kang. 2017. Personalized Cinemagraphs Using Semantic Understanding and Collaborative Learning. In Proc. ICCV.Google Scholar
    27. S. Qin, S. Kim, and R. Manduchi. 2017. Automatic skin and hair masking using fully convolutional networks. In Proc. ICME.Google Scholar
    28. Wenqi Ren, Jinshan Pan, Xiaochun Cao, and Ming-Hsuan Yang. 2017. Video Deblurring via Semantic Segmentation and Pixel-Wise Non-Linear Kernel. In Proc. ICCV.Google ScholarCross Ref
    29. Christoph Rhemann, Carsten Rother, Jue Wang, Margrit Gelautz, Pushmeet Kohli, and Pamela Rott. 2009. A Perceptually Motivated Online Benchmark for Image Matting. In Proc. CVPR.Google ScholarCross Ref
    30. Xiaoyong Shen, Xin Tao, Hongyun Gao, Chao Zhou, and Jiaya Jia. 2016. Deep Automatic Portrait Matting. In Proc. ECCV.Google ScholarCross Ref
    31. D. Singaraju and R. Vidal. 2011. Estimation of Alpha Mattes for Multiple Image Layers. IEEE Trans. Pattern Anal. Mach. Intell. 33, 7 (2011), 1295–1309. Google ScholarDigital Library
    32. Kihyuk Sohn. 2016. Improved deep metric learning with multi-class N-pair loss objective. In Proc. NIPS. Google ScholarDigital Library
    33. Yu-Wing Tai, Jiaya Jia, and Chi-Keung Tang. 2007. Soft Color Segmentation and Its Applications. IEEE Trans. Pattern Anal. Mach. Intell. 29, 9 (2007), 1520–1537. Google ScholarDigital Library
    34. Jianchao Tan, Jyh-Ming Lien, and Yotam Gingold. 2016. Decomposing Images into Layers via RGB-space Geometry. ACM Trans. Graph. 36, 1 (2016), 7:1–7:14. Google ScholarDigital Library
    35. Ning Xu, Brian Price, Scott Cohen, and Thomas Huang. 2017. Deep Image Matting. In Proc. CVPR.Google ScholarCross Ref
    36. Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. 2017. Pyramid Scene Parsing Network. In Proc. CVPR.Google ScholarCross Ref

ACM Digital Library Publication: