“Distortion-free wide-angle portraits on camera phones” by Shih, Lai and Liang

  • ©YiChang Shih, Wei-Sheng Lai, and Chia-Kai Liang

Conference:


Type:


Title:

    Distortion-free wide-angle portraits on camera phones

Session/Category Title:   Photo Science


Presenter(s)/Author(s):



Abstract:


    Photographers take wide-angle shots to enjoy expanding views, group portraits that never miss anyone, or composite subjects with spectacular scenery background. In spite of the rapid proliferation of wide-angle cameras on mobile phones, a wider field-of-view (FOV) introduces a stronger perspective distortion. Most notably, faces are stretched, squished, and skewed, to look vastly different from real-life. Correcting such distortions requires professional editing skills, as trivial manipulations can introduce other kinds of distortions. This paper introduces a new algorithm to undistort faces without affecting other parts of the photo. Given a portrait as an input, we formulate an optimization problem to create a content-aware warping mesh which locally adapts to the stereographic projection on facial regions, and seamlessly evolves to the perspective projection over the background. Our new energy function performs effectively and reliably for a large group of subjects in the photo. The proposed algorithm is fully automatic and operates at an interactive rate on the mobile platform. We demonstrate promising results on a wide range of FOVs from 70° to 120°.

References:


    1. Michael D Abràmoff, Paulo J Magalhàes, and Sunanda J Ram. 2004. Image processing with ImageJ. Biophotonics international 11, 7 (2004), 36–42.Google Scholar
    2. Sameer Agarwal, Keir Mierle, and Others. 2012. Ceres Solver. http://ceres-solver.org.Google Scholar
    3. Aseem Agarwala, Maneesh Agrawala, Michael Cohen, David Salesin, and Richard Szeliski. 2006. Photographing long scenes with multi-viewpoint panoramas. In ACM TOG, Vol 25. 853–861. Google ScholarDigital Library
    4. Dimitri P Bertsekas. 1999. Nonlinear programming.Google Scholar
    5. Duane C Brown. 1966. Decentering distortion of lenses. Photogrammetric Engineering and Remote Sensing (1966).Google Scholar
    6. Robert Carroll, Aseem Agarwala, and Maneesh Agrawala. 2010. Image warps for artistic perspective manipulation. In ACM TOG, Vol. 29. 127. Google ScholarDigital Library
    7. Robert Carroll, Maneesh Agrawal, and Aseem Agarwala. 2009. Optimizing content-preserving projections for wide-angle images. In ACM TOG, Vol. 28. 43. Google ScholarDigital Library
    8. Che-Han Chang and Yung-Yu Chuang. 2012. A line-structure-preserving approach to image resizing. In Proc. CVPR. 1075–1082. Google ScholarDigital Library
    9. Che-Han Chang, Chia-Kai Liang, and Yung-Yu Chuang. 2011. Content-aware display adaptation and interactive editing for stereoscopic images. IEEE TMM 13, 4 (2011), 589–601. Google ScholarDigital Library
    10. Che-Han Chang, Yoichi Sato, and Yung-Yu Chuang. 2014. Shape-preserving half-projective warps for image stitching. In Proc. CVPR. 3254–3261. Google ScholarDigital Library
    11. Yu-Sheng Chen and Yung-Yu Chuang. 2016. Natural image stitching with the global similarity prior. In Proc. ECCV. 186–201.Google ScholarCross Ref
    12. Song-Pei Du, Shi-Min Hu, and Ralph R Martin. 2013. Changing perspective in stereoscopic images. IEEE TVCG 19, 8 (2013), 1288–1297. Google ScholarDigital Library
    13. Claude E Duchon. 1979. Lanczos filtering in one and two dimensions. Journal of applied meteorology 18, 8 (1979), 1016–1022.Google ScholarCross Ref
    14. DxO. 2018. Correcting volume deformation with DxO ViewPoint. https://www.dxo.com/dxo-viewpoint/features/.Google Scholar
    15. Ohad Fried, Eli Shechtman, Dan B Goldman, and Adam Finkelstein. 2016. Perspective-aware manipulation of portrait photos. ACM TOG 35, 4 (2016), 128. Google ScholarDigital Library
    16. Ran Gal, Olga Sorkine, and Daniel Cohen-Or. 2006. Feature-aware texturing. Rendering Techniques 2006, 17 (2006), 2. Google ScholarDigital Library
    17. GoPro. 2017. HERO6 Black Field of View (FOV) Information. https://gopro.com/help/articles/question_answer/hero6-black-field-of-view-fov-information.Google Scholar
    18. Heng Guo, Shuaicheng Liu, Tong He, Shuyuan Zhu, Bing Zeng, and Moncef Gabbouj. 2016. Joint video stitching and stabilization from moving cameras. IEEE TIP 25, 11, 5491–5503.Google Scholar
    19. Kaiming He, Huiwen Chang, and Jian Sun. 2013. Rectangling panoramic images via warping. ACM TOG 32, 4 (2013), 79. Google ScholarDigital Library
    20. Chintan Intwala and Aseem Agarwala. 2015. Perspective warp. US Patent 9,117,253.Google Scholar
    21. Hamid Izadinia, Bryan C Russell, Ali Farhadi, Matthew D Hoffman, and Aaron Hertzmann. 2015. Deep classifiers from image tags in the wild. In Proc. 2015 Workshop on Community-Organized Multimodal Mining: Opportunities for Novel Solutions. 13–18. Google ScholarDigital Library
    22. Yoshihiro Kanamori, Nguyen Huu Cuong, and Tomoyuki Nishita. 2011. Local optimization of distortions in wide-angle images using moving least-squares. In Proceedings of the 27th Spring Conference on Computer Graphics. ACM, 51–56. Google ScholarDigital Library
    23. Johannes Kopf, Dani Lischinski, Oliver Deussen, Daniel Cohen-Or, and Michael Cohen. 2009. Locally adapted projections to reduce panorama distortions. In CGF, Vol. 28. 1083–1089. Google ScholarDigital Library
    24. Hyunjoon Lee, Eli Shechtman, Jue Wang, and Seungyong Lee. 2012. Automatic upright adjustment of photographs. In Proc. CVPR. 877–884. Google ScholarDigital Library
    25. Tommer Leyvand, Daniel Cohen-Or, Gideon Dror, and Dani Lischinski. 2008. Data-driven enhancement of facial attractiveness. ACM TOG 27, 3 (2008), 38. Google ScholarDigital Library
    26. Dongping Li, Kaiming He, Jian Sun, and Kun Zhou. 2015. A geodesic-preserving method for image warping. In Proc. CVPR. 213–221.Google Scholar
    27. Feng Liu, Michael Gleicher, Hailin Jin, and Aseem Agarwala. 2009. Content-preserving warps for 3D video stabilization. In ACM TOG, Vol. 28. ACM, 44. Google ScholarDigital Library
    28. Feng Liu, Michael Gleicher, Jue Wang, Hailin Jin, and Aseem Agarwala. 2011. Subspace video stabilization. ACM TOG 30, 1 (2011), 4. Google ScholarDigital Library
    29. Darko Pavić, Volker Schönefeld, and Leif Kobbelt. 2006. Interactive image completion with perspective correction. The Visual Computer 22, 9–11 (2006), 671–681. Google ScholarDigital Library
    30. Samsung. 2017. GalaxyS7: About Shape Correction. www.samsung.com/nz/support/mobile-devices/galaxy-s7-about-shape-correction/.Google Scholar
    31. Thomas K Sharpless, Bruno Postle, and Daniel M German. 2010. Pannini: a new projection for rendering wide angle perspective images. In Proc. 6th international conference on Computational Aesthetics in Graphics, Visualization and Imaging. 9–16. Google ScholarDigital Library
    32. John Parr Snyder. 1987. Map projections-A working manual. Vol. 1395. US Government Printing Office.Google Scholar
    33. Mahdi Abbaspour Tehrani, Aditi Majumder, and M Gopi. 2016. Correcting perceived perspective distortions using object specific planar transformations. In Proc. ICCP.Google ScholarCross Ref
    34. TensorFlow. 2018. TensorFlow-Mobile/Lite. https://www.tensorflow.org/mobile/.Google Scholar
    35. Kim H Veltman. 1986. Perspective, anamorphosis and vision. Marburger Jahrbuch für (1986).Google Scholar
    36. Dhanraj Vishwanath, Ahna R Girshick, and Martin S Banks. 2005. Why pictures look right when viewed from the wrong place. Nature neuroscience 8, 10 (2005), 1401.Google Scholar
    37. Neal Wadhwa, Rahul Garg, David Jacobs, Bryan Feldman, Nori Kanazawa, Robert Carroll, Yair Movshovitz-Attias, Jonathan Barron, Yael Pritch, and Marc Levoy. 2018. Synthetic depth-of-field with a single-camera mobile phone. ACM TOG 37, 4 (2018). Google ScholarDigital Library
    38. Yu-Shuen Wang, Chiew-Lan Tai, Olga Sorkine, and Tong-Yee Lee. 2008. Optimized scale-and-stretch for image resizing. In ACM TOG, Vol. 27. 118. Google ScholarDigital Library
    39. Jin Wei, Chen-Feng Li, Shi-Min Hu, Ralph R Martin, and Chiew-Lan Tai. 2012. Fisheye video correction. IEEE TVCG 18, 10 (2012), 1771–1783. Google ScholarDigital Library
    40. Lior Wolf, Moshe Guttmann, and Daniel Cohen-Or. 2007. Non-homogeneous content-driven video-retargeting. (2007).Google Scholar
    41. Lihi Zelnik-Manor, Gabriele Peters, and Pietro Perona. 2005. Squaring the circle in panoramas. In Proc. ICCV, Vol. 2. 1292–1299. Google ScholarDigital Library
    42. David Zhang, Qijun Zhao, and Fangmei Chen. 2011. Quantitative analysis of human facial beauty using geometric features. Pattern Recognition 44, 4 (2011), 940–950. Google ScholarDigital Library
    43. Fang-Lue Zhang, Xian Wu, Hao-Tian Zhang, Jue Wang, and Shi-Min Hu. 2016. Robust background identification for dynamic video editing. ACM TOG 35, 6 (2016), 197. Google ScholarDigital Library
    44. Zhengyou Zhang. 2000. A flexible new technique for camera calibration. IEEE Trans. PAMI 22 (2000). Google ScholarDigital Library
    45. Denis Zorin and Alan H Barr. 1995. Correction of geometric perceptual distortions in pictures. In SIGGRAPH. 257–264. Google ScholarDigital Library


ACM Digital Library Publication:



Overview Page: