“Mirror mirror: crowdsourcing better portraits” by Zhu, Agarwala, Efros, Shechtman and Wang
Conference:
Type(s):
Title:
- Mirror mirror: crowdsourcing better portraits
Session/Category Title: Digital Photography
Presenter(s)/Author(s):
Abstract:
We describe a method for providing feedback on portrait expressions, and for selecting the most attractive expressions from large video/photo collections. We capture a video of a subject’s face while they are engaged in a task designed to elicit a range of positive emotions. We then use crowdsourcing to score the captured expressions for their attractiveness. We use these scores to train a model that can automatically predict attractiveness of different expressions of a given person. We also train a cross-subject model that evaluates portrait attractiveness of novel subjects and show how it can be used to automatically mine attractive photos from personal photo collections. Furthermore, we show how, with a little bit ($5-worth) of extra crowdsourcing, we can substantially improve the cross-subject model by “fine-tuning” it to a new individual using active learning. Finally, we demonstrate a training app that helps people learn how to mimic their best expressions.
References:
1. Ailon, N. 2012. An active learning algorithm for ranking from pairwise preferences with an almost optimal query complexity. Journal of Machine Learning Research 13, 1, 137–164.
2. Albuquerque, G., Stich, T., Sellent, A., and Magnor, M. 2008. The good, the bad and the ugly: Attractive portraits from video sequences. In European Conference on Visual Media Production.
3. Altwaijry, H., and Belongie, S. 2013. Relative ranking of facial attractiveness. In IEEE Winter Conference on Applications of Computer Vision, 117–124.
4. Ambadar, Z., Cohn, J. F., and Reed, L. I. 2009. All smiles are not created equal: Morphology and timing of smiles perceived as amused, polite, and embarrassed/nervous. Journal of Nonverbal Behavior 33, 1, 17–34.Cross Ref
5. Bradley, R. A., and Terry, M. E. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika 39, 3/4, 324–345.
6. Burt, P. J., and Adelson, E. H. 1983. A multiresolution spline with application to image mosaics. ACM Transactions on Graphics 2, 4, 217–236.
7. Calder, A., Rhodes, G., Johnson, M., and Haxby, J. 2012. Oxford Handbook of Face Perception. Oxford University Press.
8. Chen, X., Bennett, P. N., Collins-Thompson, K., and Horvitz, E. 2013. Pairwise ranking aggregation in a crowdsourced setting. In ACM International Conference on Web Search and Data Mining, 193–202.
9. Dalal, N., and Triggs, B. 2005. Histograms of oriented gradients for human detection. In IEEE Conference on Computer Vision and Pattern Recognition.
10. Dibeklioglu, H., Gevers, T., and Salah, A. A. 2012. Are you really smiling at me? spontaneous versus posed enjoyment smiles. In European Conference on Computer Vision, no. 3, 525–538.
11. Du, S., Tao, Y., and Martinez, A. M. 2014. Compound facial expressions of emotion. Proceedings of the National Academy of Science.
12. Ekman, P., and Friesen, W. V. 1978. The Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press.
13. Fiss, J., Agarwala, A., and Curless, B. 2011. Candid portrait selection from video. ACM Transactions on Graphics 30, 6, 128:1–128:8.
14. Friedman, J. H. 2001. Greedy function approximation: a gradient boosting machine. Annals of Statistics, 1189–1232.
15. Gallagher, A., and Chen, T. 2008. Clothing cosegmentation for recognizing people. In IEEE Conference on Computer Vision and Pattern Recognition.
16. Gray, D., Yu, K., Xu, W., and Gong, Y. 2010. Predicting facial beauty without landmarks. In European Conference on Computer Vision. 434–447.
17. Gross, J., and Levenson, R. 1995. Emotion elicitation using films. Cognition & Emotion.
18. Gunnery, S. D., Hall, J. A., and Ruben, M. A. 2012. The deliberate duchenne smile: Individual differences in expressive control. Journal of Nonverbal Behavior 37, 1, 1–13.
19. Hariharan, B., Malik, J., and Ramanan, D. 2012. Discriminative decorrelation for clustering and classification. In European Conference on Computer Vision. 459–472.
20. Jamieson, K. G., and Nowak, R. D. 2011. Active ranking using pairwise comparisons. In Neural Information Processing Systems, 2240–2248.
21. Jiang, B., Valstar, M. F., and Pantic, M. 2011. Action unit detection using sparse appearance descriptors in space-time video volumes. In International Conference on Automatic Face & Gesture Recognition, 314–321.
22. Joshi, N., Matusik, W., Adelson, E. H., and Kriegman, D. J. 2010. Personal photo enhancement using example images. ACM Transactions on Graphics 29, 2, 1–15.
23. Kagian, A., Dror, G., Leyvand, T., Meilijson, I., Cohenor, D., and Ruppin, E. 2008. A machine learning predictor of facial attractiveness revealing human-like psychophysical biases. Vision research 48, 2, 235–43.
24. Kemelmacher-Shlizerman, I., Shechtman, E., Garg, R., and Seitz, S. M. 2011. Exploring photobios. ACM Transactions on Graphics 30, 4, 61.
25. Khosla, A., Bainbridge, W. A., Torralba, A., and Oliva, A. 2013. Modifying the memorability of face photographs. In International Conference on Computer Vision.
26. Kiapour, M. H., Yamaguchi, K., Berg, A. C., and Berg, T. L. 2014. Hipster wars: Discovering elements of fashion styles. In European Conference on Computer Vision. 472–488.
27. Krumhuber, E. G., and Manstead, A. S. R. 2009. Can duchenne smiles be feigned? new evidence on felt and false smiles. Emotion 9, 6, 807–820.Cross Ref
28. Leyvand, T., Cohen-Or, D., Dror, G., and Lischinski, D. 2008. Data-driven enhancement of facial attractiveness. ACM Transactions on Graphics 27, 3, 38:1–38:9.
29. Liang, L., and Grauman, K. 2014. Beyond comparing image pairs: Setwise active learning for relative attributes. In IEEE Conference on Computer Vision and Pattern Recognition.
30. McDuff, D., Kaliouby, R. E., and Picard, R. W. 2012. Crowdsourcing facial responses to online videos. IEEE Transactions on Affective Computing 3, 4, 456–468.
31. O’Donovan, P., Libeks, J., Agarwala, A., and Hertzmann, A. 2014. Exploratory Font Selection Using Crowdsourced Attributes. ACM Transactions on Graphics 33, 4.
32. Oosterhof, N. N., and Todorov, A. 2008. The functional basis of face evaluation. Proceedings of the National Academy of Science 105, 32, 11087–11092.Cross Ref
33. Pantic, M., and Rothkrantz, L. J. M. 2000. Automatic analysis of facial expressions: The state of the art. IEEE Transactions on Pattern Analysis and Machine Intelligence 22, 1424–1445.
34. Parikh, D., and Grauman, K. 2011. Relative Attributes. In International Conference on Computer Vision.
35. Shah, R., and Kwatra, V. 2012. All smiles: Automatic photo enhancement by facial expression analysis. In European Conference on Visual Media Production.
36. Smola, A. J., and Schölkopf, B. 2004. A tutorial on support vector regression. Statistics and computing 14, 3, 199–222.
37. Springer, I. N., Wiltfang, J., Kowalski, J. T., Russo, P. A. J., Schulze, M., Becker, S., and Wolfart, S.. 2012. Mirror, mirror on the wall: self-perception of facial beauty versus judgement by others. Journal of cranio-maxillo-facial surgery 40, 8, 773–6.Cross Ref
38. Tsukida, K., And Gupta, M. R. 2011. How to analyze paired comparison data. Tech. Rep. UWEETR-2011-0004, Dept. of Electrical Engineering, University of Washington.
39. Wang, J., and Cohen, M. F. 2005. Very low frame-rate video streaming for face-to-face teleconference. In Proceedings of the Data Compression Conference, 309–318.
40. Xiong, X., and De la Torre, F. 2013. Supervised descent method and its applications to face alignment. In IEEE Conference on Computer Vision and Pattern Recognition, 532–539.
41. Yang, F., Wang, J., Shechtman, E., Bourdev, L., and Metaxas, D. 2011. Expression flow for 3d-aware face component transfer. ACM Transactions on Graphics 30, 4, 60.
42. Zhang, L., Snavely, N., Curless, B., and Seitz, S. M. 2004. Spacetime faces: High-resolution capture for modeling and animation.
