“Designing Perceptual Puzzles by Differentiating Probabilistic Programs” by Chandra, Li, Tenenbaum and Ragan-Kelley

  • ©Kartik Chandra, Tzu-Mao Li, Joshua Tenenbaum, and Jonathan Ragan-Kelley

Conference:


Title:


    Designing Perceptual Puzzles by Differentiating Probabilistic Programs

Program Title:


    Demo Labs

Presenter(s):


Description:


    We design new visual illusions by finding “adversarial examples” for principled models of human perception — specifically, for probabilistic models, which treat vision as Bayesian inference. To perform this search efficiently, we design a differentiable probabilistic programming language, whose API exposes MCMC inference as a first-class differentiable function. We demonstrate our method by automatically creating illusions for three features of human vision: color constancy, size constancy, and face perception.

References:


    1. Chris L Baker, Rebecca Saxe, and Joshua B Tenenbaum. 2009. Action understanding as inverse planning. Cognition 113, 3 (2009), 329–349. https://dspace.mit.edu/handle/1721.1/60852
    2. Sai Bangaru, Jesse Michel, Kevin Mu, Gilbert Bernstein, Tzu-Mao Li, and Jonathan Ragan-Kelley. 2021. Systematically Differentiating Parametric Discontinuities. ACM Trans. Graph. 40, 107 (2021), 107:1–107:17. https://people.csail.mit.edu/sbangaru/projects/teg-2021/teg-2021.pdf
    3. Peter W Battaglia, Jessica B Hamrick, and Joshua B Tenenbaum. 2013. Simulation as an engine of physical scene understanding. Proceedings of the National Academy of Sciences 110, 45(2013), 18327–18332. https://www.pnas.org/content/pnas/110/45/18327.full.pdf
    4. Adrien Bousseau, Emmanuelle Chapoulie, Ravi Ramamoorthi, and Maneesh Agrawala. 2011. Optimizing environment maps for material depiction. In Computer graphics forum, Vol. 30. 1171–1180. https://dl.acm.org/doi/abs/10.1111/j.1467-8659.2011.01975.x
    5. David H Brainard and William T Freeman. 1997. Bayesian color constancy. JOSA A 14, 7 (1997), 1393–1411. http://people.csail.mit.edu/billf/publications/Bayesian_Color_Constancy.pdf
    6. Andrew Campbell, Wenlong Chen, Vincent Stimper, Jose Miguel Hernandez-Lobato, and Yichuan Zhang. 2021. A Gradient Based Strategy for Hamiltonian Monte Carlo Hyperparameter Optimization. In International Conference on Machine Learning. PMLR, 1238–1248. http://proceedings.mlr.press/v139/campbell21a/campbell21a.pdf
    7. Bob Carpenter, Andrew Gelman, Matthew D Hoffman, Daniel Lee, Ben Goodrich, Michael Betancourt, Marcus Brubaker, Jiqiang Guo, Peter Li, and Allen Riddell. 2017. Stan: A probabilistic programming language. Journal of statistical software 76, 1 (2017), 1–32. https://www.jstatsoft.org/article/view/v076i01
    8. Kartik Chandra, Chuma Kabaghe, and Gregory Valiant. 2021. Beyond Laurel/Yanny: An Autoencoder-Enabled Search for Polyperceivable Audio. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers). Association for Computational Linguistics, Online, 593–598. https://doi.org/10.18653/v1/2021.acl-short.75
    9. Ming-Te Chi, Tong-Yee Lee, Yingge Qu, and Tien-Tsin Wong. 2008. Self-Animating Images: Illusory Motion Using Repeated Asymmetric Patterns. ACM Trans. Graph. 27, 3 (aug 2008), 1–8. https://doi.org/10.1145/1360612.1360661
    10. Hung-Kuo Chu, Wei-Hsin Hsu, Niloy J Mitra, Daniel Cohen-Or, Tien-Tsin Wong, and Tong-Yee Lee. 2010. Camouflage images.ACM Trans. Graph. 29, 4 (2010), 51–1. https://dl.acm.org/doi/abs/10.1145/1833349.1778788
    11. Bo Dai, Zhen Liu, Hanjun Dai, Niao He, Arthur Gretton, Le Song, and Dale Schuurmans. 2019. Exponential Family Estimation via Adversarial Dynamics Embedding. In Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. Fox, and R. Garnett (Eds.). Vol. 32. Curran Associates, Inc.https://proceedings.neurips.cc/paper/2019/file/767d01b4bac1a1e8824c9b9f7cc79a04-Paper.pdf
    12. Yilun Du, Shuang Li, B. Joshua Tenenbaum, and Igor Mordatch. 2021. Improved Contrastive Divergence Training of Energy Based Models. In Proceedings of the 38th International Conference on Machine Learning (ICML-21). https://arxiv.org/abs/2012.01316
    13. Simon Duane, Anthony D Kennedy, Brian J Pendleton, and Duncan Roweth. 1987. Hybrid Monte Carlo. Physics letters B 195, 2 (1987), 216–222. https://archive.org/download/wikipedia-scholarly-sources-corpus/10.1016%252F0361-9230%252887%252990129-8.zip/10.1016%252F0370-2693%252887%252991197-X.pdf
    14. Frédo Durand, Maneesh Agrawala, Bruce Gooch, Victoria Interrante, Victor Ostromoukhov, and Denis Zorin. 2002. Perceptual and artistic principles for effective computer depiction. SIGGRAPH 2002 Course# 13 Notes(2002). http://people.csail.mit.edu/fredo/SIG02_ArtScience/DepictionNotes2.pdf
    15. Gamaleldin Elsayed, Shreya Shankar, Brian Cheung, Nicolas Papernot, Alexey Kurakin, Ian Goodfellow, and Jascha Sohl-Dickstein. 2018. Adversarial examples that fool both computer vision and time-limited humans. Advances in neural information processing systems 31 (2018). https://proceedings.neurips.cc/paper/2018/file/8562ae5e286544710b2e7ebe9858833b-Paper.pdf
    16. Wilson S Geisler and Daniel Kersten. 2002. Illusions, perception and Bayes. Nature neuroscience 5, 6 (2002), 508–510. https://www.cs.utexas.edu/~dana/NVGeisler2.pdf
    17. Thomas Gerig, Andreas Morel-Forster, Clemens Blumer, Bernhard Egger, Marcel Luthi, Sandro Schönborn, and Thomas Vetter. 2018. Morphable face models-an open framework. In 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018). IEEE, 75–82. https://ieeexplore.ieee.org/abstract/document/8373814
    18. Noah D Goodman, Joshua B. Tenenbaum, and The ProbMods Contributors. 2016. Probabilistic Models of Cognition. http://probmods.org/v2. Accessed: 2021-10-15.
    19. Nishad Gothoskar, Marco Cusumano-Towner, Ben Zinberg, Matin Ghavamizadeh, Falk Pollok, Austin Garrett, Josh Tenenbaum, Dan Gutfreund, and Vikash Mansinghka. 2021. 3DP3: 3D Scene Perception via Probabilistic Programming. Advances in Neural Information Processing Systems 34 (2021). https://proceedings.neurips.cc/paper/2021/file/4fc66104f8ada6257fa55f29a2a567c7-Paper.pdf
    20. Aaron Hertzmann. 2020. Why do line drawings work? a realism hypothesis. Perception 49, 4 (2020), 439–451. https://arxiv.org/abs/2002.06260
    21. Inbar Huberman and Raanan Fattal. 2015. Reducing Lateral Visual Biases in Displays. Computer Graphics Forum (CGF). https://www.cs.huji.ac.il/w~raananf/projects/lateral_biases/
    22. Daniel Kersten, Pascal Mamassian, and Alan Yuille. 2004. Object perception as Bayesian inference. Annu. Rev. Psychol. 55(2004), 271–304. http://mamassian.free.fr/papers/KerstenMamassianYuille04.pdf
    23. Erum Arif Khan, Erik Reinhard, Roland W. Fleming, and Heinrich H. Bülthoff. 2006. Image-Based Material Editing. ACM Trans. Graph. 25, 3 (jul 2006), 654–663. https://doi.org/10.1145/1141911.1141937
    24. Diederik P Kingma and Max Welling. 2014. Auto-Encoding Variational Bayes. arxiv:1312.6114 [stat.ML] https://arxiv.org/pdf/1312.6114.pdf
    25. David C Knill and Whitman Richards. 1996. Perception as Bayesian inference. Cambridge University Press.
    26. Tejas D Kulkarni, Pushmeet Kohli, Joshua B Tenenbaum, and Vikash Mansinghka. 2015. Picture: A probabilistic programming language for scene perception. In Proceedings of the IEEE conference on computer vision and pattern recognition. 4390–4399. https://mrkulk.github.io/www_cvpr15/1999.pdf
    27. Rosa Lafer-Sousa, Katherine L Hermann, and Bevil R Conway. 2015. Striking individual differences in color perception uncovered by ‘The Dress’ photograph. Current Biology 25, 13 (2015), R545–R546. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4921196/pdf/nihms686513.pdf
    28. Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. 2019. Soft rasterizer: A differentiable renderer for image-based 3D reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 7708–7717. https://arxiv.org/pdf/1904.01786.pdf
    29. Margaret S Livingstone. 2000. Is it warm? Is it real? Or just low spatial frequency?Science 290, 5495 (2000), 1299–1299. https://livingstone.hms.harvard.edu/sites/livingstone.hms.harvard.edu/files/publications/2000_November17.%20Livingstone.%20Is%20It%20Warm_%20Is%20it%20Real_%20Or%20Just%20Low%20Spatial%20Frequency_.pdf
    30. Li-Qian Ma, Kun Xu, Tien-Tsin Wong, Bi-Ye Jiang, and Shi-Min Hu. 2013. Change blindness images. IEEE transactions on visualization and computer graphics 19, 11(2013), 1808–1819. https://cg.cs.tsinghua.edu.cn/papers/TVCG-2013-changeblindness.pdf
    31. Dougal Maclaurin, David Duvenaud, and Ryan Adams. 2015. Gradient-based hyperparameter optimization through reversible learning. In International conference on machine learning. PMLR, 2113–2122. http://proceedings.mlr.press/v37/maclaurin15.pdf
    32. Pascal Mamassian, Michael Landy, and Laurence T Maloney. 2002. Bayesian modelling of visual perception. Probabilistic models of the brain(2002), 13–36. http://mamassian.free.fr/papers/mamassian_mit02.pdf
    33. Vikash K Mansinghka, Tejas D Kulkarni, Yura N Perov, and Josh Tenenbaum. 2013. Approximate Bayesian Image Interpretation using Generative Probabilistic Graphics Programs. In Advances in Neural Information Processing Systems, C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger (Eds.). Vol. 26. Curran Associates, Inc.https://proceedings.neurips.cc/paper/2013/file/fa14d4fe2f19414de3ebd9f63d5c0169-Paper.pdf
    34. David Marr. 1982. Vision. W. H. Freeman and Company. https://mitpress.mit.edu/books/vision
    35. Shakir Mohamed, Mihaela Rosca, Michael Figurnov, and Andriy Mnih. 2020. Monte Carlo Gradient Estimation in Machine Learning.J. Mach. Learn. Res. 21, 132 (2020), 1–62. https://arxiv.org/pdf/1906.10652.pdf
    36. Hadi Mohasel Afshar and Justin Domke. 2015. Reflection, Refraction, and Hamiltonian Monte Carlo. In Advances in Neural Information Processing Systems, C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (Eds.). Vol. 28. Curran Associates, Inc.https://proceedings.neurips.cc/paper/2015/file/8303a79b1e19a194f1875981be5bdb6f-Paper.pdf
    37. Radford M Neal 2011. MCMC using Hamiltonian dynamics. Handbook of Markov chain Monte Carlo 2, 11 (2011), 2. https://arxiv.org/pdf/1206.1901.pdf
    38. Yu Okano, Shogo Fukushima, Masahiro Furukawa, and Hiroyuki Kajimoto. 2010. Embedded Motion: Generating the Perception of Motion in Peripheral Vision. In ACM SIGGRAPH ASIA 2010 Posters (Seoul, Republic of Korea) (SA ’10). Association for Computing Machinery, New York, NY, USA, Article 41, 1 pages. https://doi.org/10.1145/1900354.1900400
    39. Aude Oliva, Antonio Torralba, and Philippe G Schyns. 2006. Hybrid images. ACM Transactions on Graphics (TOG) 25, 3 (2006), 527–532. https://stanford.edu/class/ee367/reading/OlivaTorralb_Hybrid_Siggraph06.pdf
    40. Tobias Ritschel, Kaleigh Smith, Matthias Ihrke, Thorsten Grosch, Karol Myszkowski, and Hans-Peter Seidel. 2008. 3D Unsharp Masking for Scene Coherent Enhancement. Vol. 27. Association for Computing Machinery, New York, NY, USA, 1–8. https://doi.org/10.1145/1360612.1360689
    41. Tim Salimans, Diederik Kingma, and Max Welling. 2015. Markov chain Monte Carlo and variational inference: Bridging the gap. In International Conference on Machine Learning. PMLR, 1218–1226. http://proceedings.mlr.press/v37/salimans15.pdf
    42. Roger N Shepard. 1994. Perceptual-cognitive universals as reflections of the world. Psychonomic Bulletin & Review 1, 1 (1994), 2–28. http://ruccs.rutgers.edu/images/personal-zenon-pylyshyn/docs/transfers/shepard_space_bbs2001.pdf
    43. Fabian H. Sinz, Xaq Pitkow, Jacob Reimer, Matthias Bethge, and Andreas S. Tolias. 2019. Engineering a Less Artificial Intelligence. Neuron 103, 6 (2019), 967–979. https://doi.org/10.1016/j.neuron.2019.08.034
    44. Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2014. Intriguing properties of neural networks. In International Conference on Learning Representations. http://arxiv.org/abs/1312.6199
    45. Corey Toler-Franklin, Adam Finkelstein, and Szymon Rusinkiewicz. 2007. Illustration of complex real-world objects using images with normals. In Proceedings of the 5th international symposium on Non-photorealistic animation and rendering. 111–119. https://pixl.cs.princeton.edu/pubs/Toler-Franklin_2007_IOC/rgbn.pdf
    46. Nikolaus F Troje and Ulrike Siebeck. 1998. Illumination-induced apparent shift in orientation of human heads. Perception 27, 6 (1998), 671–680. https://journals.sagepub.com/doi/pdf/10.1068/p270671#also-see-slack
    47. Arash Vahdat, Evgeny Andriyash, and William Macready. 2020. Undirected graphical models as approximate posteriors. In International Conference on Machine Learning. PMLR, 9680–9689. http://proceedings.mlr.press/v119/vahdat20a/vahdat20a.pdf
    48. Jan-Willem van de Meent, Brooks Paige, Hongseok Yang, and Frank Wood. 2018. An introduction to probabilistic programming. arXiv preprint arXiv:1809.10756(2018). https://arxiv.org/pdf/1809.10756
    49. Pascal Wallisch. 2017. Illumination assumptions account for individual differences in the perceptual interpretation of a profoundly ambiguous stimulus in the color domain:“The Dress”. Journal of Vision 17, 4 (2017), 5–5. https://jov.arvojournals.org/article.aspx?articleid=2617976&mbid=synd_yahoostyle
    50. Pascal Wallisch and Michael Karlovich. 2019. Disagreeing about Crocs and socks: Creating profoundly ambiguous color displays. arXiv preprint arXiv:1908.05736(2019). https://arxiv.org/pdf/1908.05736.pdf
    51. David R. Walton, Rafael Kuffner Dos Anjos, Sebastian Friston, David Swapp, Kaan Akşit, Anthony Steed, and Tobias Ritschel. 2021. Beyond Blur: Real-Time Ventral Metamers for Foveated Rendering. ACM Trans. Graph. 40, 4, Article 48 (jul 2021), 14 pages. https://doi.org/10.1145/3450626.3459943
    52. Yair Weiss, Eero P Simoncelli, and Edward H Adelson. 2002. Motion illusions as optimal percepts. Nature neuroscience 5, 6 (2002), 598–604. https://www.nature.com/articles/nn858
    53. Christoph Witzel and Matteo Toscani. 2020. How to make a #theDress. JOSA A 37, 4 (2020), A202–A211. https://www.osapublishing.org/josaa/abstract.cfm?uri=josaa-37-4-A202
    54. Frank Wood, Jan Willem van de Meent, and Vikash Mansinghka. 2014. A New Approach to Probabilistic Programming Inference. In Proceedings of the 17th International conference on Artificial Intelligence and Statistics. 1024–1032. https://probprog.github.io/anglican/assets/pdf/wood-aistats-2014.pdf
    55. Guodong Zhang, Kyle Hsu, Jianing Li, Chelsea Finn, and Roger B Grosse. 2021. Differentiable annealed importance sampling and the perils of gradient noise. Advances in Neural Information Processing Systems 34 (2021). https://proceedings.neurips.cc/paper/2021/file/a1a609f1ac109d0be28d8ae112db1bbb-Paper.pdf
    56. Yuan Zhou, Bradley J. Gram-Hansen, Tobias Kohn, Tom Rainforth, Hongseok Yang, and Frank Wood. 2019. LF-PPL: A Low-Level First Order Probabilistic Programming Language for Non-Differentiable Models. In Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics(Proceedings of Machine Learning Research, Vol. 89), Kamalika Chaudhuri and Masashi Sugiyama (Eds.). PMLR, 148–157. https://proceedings.mlr.press/v89/zhou19b.html
    57. Zhenglong Zhou and Chaz Firestone. 2019. Humans can decipher adversarial images. Nature communications 10, 1 (2019), 1–9. https://www.nature.com/articles/s41467-019-08931-6
    58. David M Zoltowski, Diana Cai, and Ryan P Adams. 2020. Slice Sampling Reparameterization Gradients. In Third Symposium on Advances in Approximate Bayesian Inference. https://openreview.net/pdf?id=cT_RMSqVf4

ACM Digital Library Publication:


Overview Page:


Type: