Up to Speed on Deep Learning: June 11–18 Update

Sharing some of the latest research, announcements, and resources on deep learning.

By Isaac Madan (email)

Continuing our series of deep learning updates, we pulled together some of the awesome resources that have emerged since our last post. In case you missed it, here are our past updates: June (part 1, part 2), May, April (part 1, part 2), March part 1, February, November, September part 2 & October part 1, September part 1, August (part 1, part 2), July (part 1, part 2), June, and the original set of 20+ resources we outlined in April 2016. As always, this list is not comprehensive, so let us know if there’s something we should add, or if you’re interested in discussing this area further.

Research & Announcements

Learning to Speak via Interaction by Baidu Research. Teaching an AI agent to speak by interacting with a virtual agent. This represents an advancement in more closely replicating how humans learn, as well as advancing our goal to demonstrate general artificial intelligence. Our AI agent learns to speak in an interactive way similar to a baby. In contrast, the conventional approach relies on supervised training using a large corpus of pre-collected training set, which is static and makes it hard to capture the interactive nature within the process of language learning. Original paper here.

Deep Shimon: Robot that composes its own music by Mason Britan of Georgia Tech. The robot Shimon composes and performs his first deep learning driven piece. A recurrent deep neural network is trained on a large database of classical and jazz music. Based on learned semantic relationships between musical units in this dataset, Shimon generates and performs a new musical piece. Video here.

Curiosity-driven Exploration by Self-supervised Prediction by Pathak et al. UC Berkeley researchers demonstrate artificial curiosity via an intrinsic curiosity model to control a virtual agent in a video game and understand its environment faster — which can accelerate problem solving. Original paper here and video here.

Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour by Facebook Research. Deep learning benefits from massive data sets, but this means long training times that slow down development. Using commodity hardware, our implementation achieves ∼90% scaling efficiency when moving from 8 to 256 GPUs. This system enables us to train visual recognition models on internet-scale data with high efficiency. Original paper here.

Resources

A gentle introduction to deep learning with TensorFlow by Michelle Fullwood at PyCon 2017. This talk aims to gently bridge the divide by demonstrating how deep learning operates on core machine learning concepts and getting attendees started coding deep neural networks using Google’s TensorFlow library. 41 minute video. Slides here and GitHub here.

Deep Reinforcement Learning Demystified (Episode 0) by Moustafa Alzantot. Basic description of what reinforcement learning is and provide examples for where it can be used. Cover the essential terminologies for reinforcement learning and provide a quick tutorial about OpenAI gym.

Neural Networks and Deep Learning by Michael Nielsen. Free online book that introduces neural networks and deep learning.

You can probably use deep learning even if your data isn’t that big by Andrew Beam. Article argues and explains how you can still use deep learning in (some) small data settings, if you train your model carefully. In response to Don’t use deep learning your data isn’t that big by Jeff Leek.

Posting on ArXiv is good, flag planting notwithstanding by Yann LeCun. In response to, and refuting, An Adversarial Review of “Adversarial Generation of Natural Language” by Yoav Goldberg of Bar Ilan University, which takes issue with deep learning researchers publishing aggressively on Arxiv.

Tutorials & Data

Computational Neuroscience Coursera course by University of Washington. Starts July 3, enroll now. Learn how the brain processes information. This course provides an introduction to basic computational methods for understanding what nervous systems do and for determining how they function. We will explore the computational principles governing various aspects of vision, sensory-motor control, learning, and memory.

Core ML and Vision: Machine Learning in iOS 11 Tutorial by Audrey Tam. iOS 11 introduces two new frameworks related to machine learning, Core ML and Vision. This tutorial walks you through how to use these new APIs and build a scene classifier.

Deep Learning CNN’s in Tensorflow with GPUs by Cole Murray. In this tutorial, you’ll learn the architecture of a convolutional neural network (CNN), how to create a CNN in Tensorflow, and provide predictions on labels of images. Finally, you’ll learn how to run the model on a GPU so you can spend your time creating better models, not waiting for them to converge.

Open-sourced Kinetics data set by Google DeepMind. Annotated data set of human actions — things like playing instruments, shaking hands, and hugging. Kinetics is a large-scale, high-quality dataset of YouTube video URLs which include a diverse range of human focused actions. The dataset consists of approximately 300,000 video clips, and covers 400 human action classes with at least 400 video clips for each action class.

Let’s evolve a neural network with a genetic algorithm by Matt Harvey of Coastline Automation. Applying a genetic algorithm to evolve a network with the goal of achieving optimal hyperparameters in a fraction of the time required to do a brute force search.


By Isaac Madan. Isaac is an investor at Venrock (email). If you’re interested in deep learning or there are resources I should share in a future newsletter, I’d love to hear from you.

Up to Speed on Deep Learning: June Update

Sharing some of the latest research, announcements, and resources on deep learning.

By Isaac Madan (email)

Continuing our series of deep learning updates, we pulled together some of the awesome resources that have emerged since our last post. In case you missed it, here are our past updates: May, April part 2, April part 1, March part 1, February, November, September part 2 & October part 1, September part 1, August part 2, August part 1, July part 2, July part 1, June, and the original set of 20+ resources we outlined in April 2016. As always, this list is not comprehensive, so let us know if there’s something we should add, or if you’re interested in discussing this area further.

Continue reading “Up to Speed on Deep Learning: June Update”

Up to Speed on Deep Learning: May Update

Sharing some of the latest research, announcements, and resources on deep learning.

By Isaac Madan (email)

Continuing our series of deep learning updates, we pulled together some of the awesome resources that have emerged since our last post. In case you missed it, here are our past updates: April part 2, April part 1, March part 1, February, November, September part 2 & October part 1, September part 1, August part 2, August part 1, July part 2, July part 1, June, and the original set of 20+ resources we outlined in April 2016. As always, this list is not comprehensive, so let us know if there’s something we should add, or if you’re interested in discussing this area further.

Continue reading “Up to Speed on Deep Learning: May Update”

Getting Up to Speed on Deep Learning

By Isaac Madan and David Dindi

For good reason, deep learning is increasingly capturing mainstream attention. Just recently, on March 15th, Google DeepMind’s AlphaGo AI — technology based on deep neural networks — beat Lee Sedol, one of the world’s best Go players, in a professional Go match.

Behind the scenes, deep learning is an active, fast-paced research area that’s proliferating quickly among some of the world’s most innovative companies. We are asked frequently about our favorite resources to get up to speed on deep learning and follow its rapid developments. As such, we’ve outlined below some of our favorite resources. While certainly not comprehensive, there’s a lot here, and we’ll continue to update this list — if there’s something we should add, let us know.

Structured Resources

Deep Learning by Ian Goodfellow, Yoshua Bengio and Aaron Courville (2016). A comprehensive and in-depth book on machine learning and deep learning core concepts.

Course notes from Stanford CS 231N: Convolutional Neural Networks for Visual Recognition. This course is a deep dive into details of neural network architectures with a focus on learning end-to-end models for these tasks, particularly image classification. During the 10-week course, students will learn to implement, train and debug their own neural networks and gain a detailed understanding of cutting-edge research in computer vision.

Course notes from Stanford CS 224D: Deep Learning for Natural Language Processing. In this class, students will learn to understand, implement, train, debug, visualize and potentially invent their own neural network models for a variety of language understanding tasks.

Blogs, Papers, and Articles

Deep Learning in a Nutshell by Tim Dettmers, via NVidia (2015). These articles are digestible and do not rely heavily on math.

  • Part 1: A gentle introduction to deep learning that covers core concepts and vocabulary.
  • Part 2: History of deep learning and methods of training deep learning architectures quickly and efficiently.
  • Part 3: Sequence learning with a focus on natural language processing.

Podcast with Yoshua Bengio: The Rise of Neural Networks and Deep Learning in Our Everyday Lives. An exciting overview of the power of neural networks as well as their current influence and future potential.

Deep learning reading list. A thorough list of academic survey papers on the subjects of reinforcement learning, computer vision, NLP & speech, disentangling factors, transfer learning, practical tricks, sparse coding, foundation theory, feedforward networks, large scale deep learning, recurrent networks, hyper parameters, optimization, and unsupervised feature learning.

Christopher Olah’s blog. Christopher has in-depth, well-explained articles with great visuals on neural networks, visualization, and convolutional neural networks.

Adrian Coyler’s blog. Adrian selects and reviews an interesting/influential/important paper from the world of CS every weekday morning.

Academic papers & presentations:

Community

Deep learning Google Group. Where deep learning enthusiasts and researchers hangout and share latest news.

Deep learning research groups. A list of many of the academic and industry labs focused on deep learning.

San Francisco AI meetup. A local meetup for AI enthusiasts and researchers that we’re involved in. Pieter Abbeel will be speaking on April 28, and Vinod Khosla on May 5.

Conferences:

  • International Conference on Learning Representations. May 2–4, 2016 in the Caribe Hilton, San Juan, Puerto Rico. Despite the importance of representation learning to machine learning and to application areas such as vision, speech, audio and NLP, there was no venue for researchers who share a common interest in this topic. The goal of ICLR has been to help fill this void. Yoshua Bengio & Yann Lecun are General Chairs.
  • International Conference on Machine Learning. June 19-24, 2016 in New York City, NY. ICML is the leading international machine learning conference and is supported by the International Machine Learning Society (IMLS).
  • Conference on Neural Information Processing Systems (NIPS).December 5–10, 2016 in Barcelona, Spain. A single-track machine learning and computational neuroscience conference that includes invited talks, demonstrations and oral and poster presentations of refereed papers.
  • GPU Technology Conference (GTC). April 4–7, 2016 in San Jose, CA; there are others later throughout the year in other countries. Presented by NVIDIA, GTC is comprised of the annual conference, year-long webinar series, and workshops that connect the global community of developers, researchers, and scientists through unique educational and networking opportunities.

Implementation

Deep Learning Frameworks in VentureBeat (2015). An overview of major deep learning libraries, as of December 2015.

TensorFlow neural network playground. Play with neural networks visually in your browser to get a feel for what they are and what they do.

TensorFlow tutorial. Google’s tutorial that explains TensorFlow and MNIST, as well as the basics of machine learning and deep learning networks. This is in Python.

OpenAI Gym. A toolkit for developing and comparing reinforcement learning algorithms. It supports teaching agents everything from walking to playing games like Pong or Go.

Neon. Nervana System’s fast Python-based deep learning library. Tutorialshere.

Debugging neural networks by Russell Stewart. Neural networks are hard to debug and this affects the learning curve involved in implementing deep learning. Russell offers some great insight.

Theano. Numerical computation library for Python (faster and more mature than TensorFlow).

Lasagne. Lightweight Python library for deep learning (built on Theano).

Caffe. Deep learning framework.

Model Zoo. Pretrained Caffe models for a variety of tasks.


By Isaac Madan and David Dindi. Isaac is an investor at Venrock (email). David is a grad student at Stanford and TA for CS 224D, mentioned above (email). If you’re interested in deep learning or working on something in this area, we’d love to hear from you.

Subscribe to our email newsletter here.

Requests for Startups is a newsletter of entrepreneurial ideas & perspectives by investors, operators, and influencers. If you think there’s someone we should feature in an upcoming issue, nominate them by sending Isaac an email.