“CNN Explainer: Learning Convolutional Neural Networks with Interactive Visualization” by Wang, Turko, Shaikh, Park, Das, et al. …
Conference:
Type(s):
Interest Area:
- Research / Education and Scientific Visualization
Title:
- CNN Explainer: Learning Convolutional Neural Networks with Interactive Visualization
Session/Category Title: TVCG Session on Data Visualization
Presenter(s)/Author(s):
Abstract:
Deep learning’s great success motivates many practitioners and students to learn about this exciting technology. However, it is often challenging for beginners to take their first step due to the complexity of understanding and applying deep learning. We present CNN Explainer, an interactive visualization tool designed for non-experts to learn and examine convolutional neural networks (CNNs), a foundational deep learning model architecture. Our tool addresses key challenges that novices face while learning about CNNs, which we identify from interviews with instructors and a survey with past students. CNN Explainer tightly integrates a model overview that summarizes a CNN’s structure, and on-demand, dynamic visual explanation views that help users understand the underlying components of CNNs. Through smooth transitions across levels of abstraction, our tool enables users to inspect the interplay between low-level mathematical operations and high-level model structures. A qualitative user study shows that CNN Explainer helps users more easily understand the inner workings of CNNs, and is engaging and enjoyable to use. We also derive design lessons from our study. Developed using modern web technologies, CNN Explainer runs locally in users’ web browsers without the need for installation or specialized hardware, broadening the public’s education access to modern deep learning techniques.
References:
[1] Tiny ImageNet Visual Recognition Challenge. https: //tiny-imagenet.herokuapp.com, 2015.
[2] Backpropagation Algorithm. https:// developers-dot-devsite-v2-prod.appspot.com/ machine-learning/crash-course/backprop-scroll, 2018.
[3] TensorSpace.js: Neural Network 3D Visualization Framework. https: //tensorspace.org, 2018.
[4] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, M. Kudlur, J. Levenberg, R. Monga, S. Moore, D. G. Murray, B. Steiner, P. Tucker, V. Vasudevan, P. Warden, M. Wicke, Y. Yu, and X. Zheng. TensorFlow: A System for Large-Scale Machine Learning. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI), pp. 265?283. Savannah, GA, USA, Nov. 2016.
[5] A. Bilal, A. Jourabloo, M. Ye, X. Liu, and L. Ren. Do Convolutional Neural Networks Learn Class Hierarchy? IEEE Transactions on Visualization and Computer Graphics, 24(1):152?162, Jan. 2018.
[6] M. Bostock, V. Ogievetsky, and J. Heer. D3 Data-Driven Documents. IEEE Transactions on Visualization and Computer Graphics, 17(12):2301?2309, Dec. 2011.
[7] M. H. Brown. Algorithm Animation. MIT Press, Cambridge, MA, USA, 1988.
[8] M. D. Byrne, R. Catrambone, and J. T. Stasko. Evaluating animations as student aids in learning computer algorithms. Computers & Education, 33(4):253?278, Dec. 1999.
[9] M. Carney, B. Webster, I. Alvarado, K. Phillips, N. Howell, J. Griffith, J. Jongejan, A. Pitaru, and A. Chen. Teachable Machine: Approachable Web-Based Tool for Exploring Machine Learning Classification. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems, CHI ?20. ACM, Honolulu, HI, USA, 2020.
[10] S. Carter and M. Nielsen. Using Artificial Intelligence to Augment Human Intelligence. Distill, 2(12), Dec. 2017.
[11] M. Conlen, A. Kale, and J. Heer. Capture & Analysis of Active Reading Behaviors for Interactive Articles on the Web. Computer Graphics Forum, 38(3):687?698, June 2019.
[12] N. Das, H. Park, Z. J. Wang, F. Hohman, R. Firstman, E. Rogers, and D. H. Chau. Massif: Interactive interpretation of adversarial attacks on deep learning. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems, CHI ?20. ACM, Honolulu, HI, USA, 2020.
[13] E. Fouh, M. Akbar, and C. A. Shaffer. The Role of Visualization in Computer Science Education. Computers in the Schools, 29(1-2):95?117, Jan. 2012.
[14] D. Galles. Data structure visualizations, 2006.
[15] R. Garcia, A. C. Telea, B. Castro da Silva, J. T?rresen, and J. L. Dihl Comba. A task-and-technique centered survey on visual analytics for deep learning model engineering. Computers & Graphics, 77:30?49, Dec. 2018.
[16] S. Grissom, M. F. McNally, and T. Naps. Algorithm visualization in CS education: Comparing levels of student engagement. In Proceedings of the 2003 ACM Symposium on Software Visualization – SoftVis ?03, pp. 87?94. San Diego, CA, USA, 2003.
[17] J. Gu, Z. Wang, J. Kuen, L. Ma, A. Shahroudy, B. Shuai, T. Liu, X. Wang, G. Wang, J. Cai, and T. Chen. Recent advances in convolutional neural networks. Pattern Recognition, 77:354?377, May 2018.
[18] P. J. Guo. Online python tutor: Embeddable web-based program visualization for cs education. In Proceeding of the 44th ACM Technical Symposium on Computer Science Education – SIGCSE ?13, pp. 579?584. ACM Press, Denver, CO, USA, 2013.
[19] S. Hansen, N. Narayanan, and M. Hegarty. Designing Educationally Effective Algorithm Visualizations. Journal of Visual Languages & Computing, 13(3):291?317, June 2002.
[20] A. W. Harley. An Interactive Node-Link Visualization of Convolutional Neural Networks. In Advances in Visual Computing, vol. 9474, pp. 867? 877. Springer International Publishing, 2015.
[21] K. He, X. Zhang, S. Ren, and J. Sun. Deep Residual Learning for Image Recognition. arXiv:1512.03385 [cs], Dec. 2015.
[22] S. Hochreiter and J. Schmidhuber. Long Short-Term Memory. Neural Computation, 9(8):1735?1780, Nov. 1997.
[23] F. Hohman, M. Kahng, R. Pienta, and D. H. Chau. Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers. IEEE Transactions on Visualization and Computer Graphics, 25(8):2674?2693, Aug. 2019.
[24] F. Hohman, H. Park, C. Robinson, and D. H. Chau. Summit: Scaling Deep Learning Interpretability by Visualizing Activation and Attribution Summarizations. IEEE Transactions on Visualization and Computer Graphics, 26(1):1096?1106, Jan. 2020.
[25] C. Hundhausen and S. Douglas. Using visualizations to learn algorithms: Should students construct their own, or view an expert?s? In Proceeding 2000 IEEE International Symposium on Visual Languages, pp. 21?28. IEEE Comput. Soc, Seattle, WA, USA, 2000.
[26] C. D. Hundhausen, S. A. Douglas, and J. T. Stasko. A Meta-Study of Algorithm Visualization Effectiveness. Journal of Visual Languages & Computing, 13(3):259?290, June 2002.
[27] M. Kahng, P. Y. Andrews, A. Kalro, and D. H. Chau. ActiVis: Visual Exploration of Industry-Scale Deep Neural Network Models. IEEE Transactions on Visualization and Computer Graphics, 24(1):88?97, Jan. 2018.
[28] M. Kahng and D. H. Chau. How Does Visualization Help People Learn Deep Learning? Evaluation of GAN Lab. In IEEE VIS 2019 Workshop on EValuation of Interactive VisuAl Machine Learning Systems, Oct. 2019.
[29] M. Kahng, N. Thorat, D. H. Chau, F. B. Viegas, and M. Wattenberg. GAN Lab: Understanding Complex Deep Generative Models using Interactive Visual Experimentation. IEEE Transactions on Visualization and Computer Graphics, 25(1):310?320, Jan. 2019.
[30] A. Karpathy. ConvNetJS MNIST demo, 2016.
[31] A. Karpathy. CS231n Convolutional Neural Networks for Visual Recognition, 2016.
[32] C. Kehoe, J. Stasko, and A. Taylor. Rethinking the evaluation of algorithm animations as learning aids: An observational study. International Journal of Human-Computer Studies, 54(2):265?284, Feb. 2001.
[33] Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. Nature, 521(7553):436?444, May 2015.
[34] M. Liu, S. Liu, H. Su, K. Cao, and J. Zhu. Analyzing the Noise Robustness of Deep Neural Networks. In 2018 IEEE Conference on Visual Analytics Science and Technology (VAST), pp. 60?71. IEEE, Berlin, Germany, Oct. 2018.
[35] M. Liu, J. Shi, K. Cao, J. Zhu, and S. Liu. Analyzing the Training Processes of Deep Generative Models. IEEE Transactions on Visualization and Computer Graphics, 24(1):77?87, Jan. 2018.
[36] M. Liu, J. Shi, Z. Li, C. Li, J. Zhu, and S. Liu. Towards Better Analysis of Deep Convolutional Neural Networks. IEEE Transactions on Visualization and Computer Graphics, 23(1):91?100, Jan. 2017.
[37] S. Liu, D. Maljovec, B. Wang, P.-T. Bremer, and V. Pascucci. Visualizing High-Dimensional Data: Advances in the Past Decade. IEEE Transactions on Visualization and Computer Graphics, 23(3):1249?1268, Mar. 2017.
[38] A. L. Maas, A. Y. Hannun, and A. Y. Ng. Rectifier nonlinearities improve neural network acoustic models. In ICML Workshop on Deep Learning for Audio, Speech and Language Processing, 2013.
[39] A. Madsen. Visualizing memorization in RNNs. Distill, 4(3):10.23915/distill.00016, Mar. 2019.
[40] R. E. Mayer and R. B. Anderson. Animations need narrations: An experimental test of a dual-coding hypothesis. Journal of Educational Psychology, 83(4):484?490, 1991.
[41] T. L. Naps, J. R. Eagan, and L. L. Norton. JHAVE?an environment to ? actively engage students in Web-based algorithm visualizations. ACM SIGCSE Bulletin, 32(1):109?113, Mar. 2000.
[42] T. L. Naps, G. Ro?ling, V. Almstrum, W. Dann, R. Fleischer, C. Hund- ? hausen, A. Korhonen, L. Malmi, M. McNally, S. Rodger, and J. A. ? Velazquez-Iturbide. Exploring the Role of Visualization and Engage- ? ment in Computer Science Education. SIGCSE Bull., 35(2):131?152, June 2002.
[43] A. P. Norton and Y. Qi. Adversarial-Playground: A visualization suite showing how adversarial examples fool deep learning. In 2017 IEEE Symposium on Visualization for Cyber Security (VizSec), pp. 1?4. IEEE, Phoenix, AZ, USA, Oct. 2017.
[44] C. Olah. Neural Networks, Manifolds, and Topology, June 2014.
[45] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, pp. 8024?8035. 2019.
[46] N. Pezzotti, T. Hollt, J. Van Gemert, B. P. Lelieveldt, E. Eisemann, and A. Vilanova. DeepEyes: Progressive Visual Analytics for Designing Deep Neural Networks. IEEE Transactions on Visualization and Computer Graphics, 24(1):98?108, Jan. 2018.
[47] D. Schweitzer and W. Brown. Interactive visualization for the active learning classroom. ACM SIGCSE Bulletin, 39(1):208, Mar. 2007.
[48] C. A. Shaffer, M. L. Cooper, A. J. D. Alon, M. Akbar, M. Stewart, S. Ponce, and S. H. Edwards. Algorithm Visualization: The State of the Field. ACM Transactions on Computing Education, 10(3):1?22, Aug. 2010.
[49] K. Simonyan and A. Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv:1409.1556 [cs], Apr. 2015.
[50] D. Smilkov, S. Carter, D. Sculley, F. B. Viegas, and M. Wattenberg. Direct- ? Manipulation Visualization of Deep Networks. arXiv:1708.03788, Aug. 2017.
[51] D. Smilkov, N. Thorat, Y. Assogba, A. Yuan, N. Kreeger, P. Yu, K. Zhang, S. Cai, E. Nielsen, D. Soergel, S. Bileschi, M. Terry, C. Nicholson, S. N. Gupta, S. Sirajuddin, D. Sculley, R. Monga, G. Corrado, F. B. Viegas, ? and M. Wattenberg. TensorFlow.js: Machine Learning for the Web and Beyond. arXiv:1901.05350 [cs], Feb. 2019.
[52] J. T. Stasko. Using student-built algorithm animations as learning aids. ACM SIGCSE Bulletin, 29(1):25?29, Mar. 1997.
[53] E. Stevens, L. Antiga, and T. Viehmann. Deep Learning with PyTorch. O?Reilly Media, 2019.
[54] H. Strobelt, S. Gehrmann, H. Pfister, and A. M. Rush. LSTMVis: A Tool for Visual Analysis of Hidden State Dynamics in Recurrent Neural Networks. IEEE Transactions on Visualization and Computer Graphics, 24(1):667?676, Jan. 2018.
[55] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, ?. Kaiser, and I. Polosukhin. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 6000?6010, 2017.
[56] J. Wang, L. Gou, H. Yang, and H.-W. Shen. GANViz: A Visual Analytics Approach to Understand the Adversarial Game. IEEE Transactions on Visualization and Computer Graphics, 24(6):1905?1917, June 2018.
[57] Z. J. Wang, R. Turko, O. Shaikh, H. Park, N. Das, F. Hohman, M. Kahng, and D. H. Chau. CNN 101: Interactive visual learning for convolutional neural networks. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems, CHI ?20. ACM, Honolulu, HI, USA, 2020.
[58] J. Yosinski, J. Clune, A. Nguyen, T. Fuchs, and H. Lipson. Understanding Neural Networks Through Deep Visualization. In ICML Deep Learning Workshop, 2015.
[59] C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals. Understanding deep learning requires rethinking generalization. In 5th International Conference on Learning Representations (ICLR), Toulon, France, Conference Track Proceedings, 2017.
