Free online resources to study reinforcement learning and deep RL

dodo · May 10, 2020

Over the last few years and months, I gathered some material that I used to study reinforcement learning and deep reinforcement learning, mostly by myself, as part of my PhD formation.

I hope these links can help others as much as they’ve helped me. The list also serves the purpose of remembering me that this material exists, since memory loss is a thing during a PhD.

Books and surveys

The current textbook for the area of reinforcement learning in general is

R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed. Cambridge, Mass: The MIT Press, 2018.

It is freely and legally available here and you can find my name in the errata.

A textual reference for the overall area of deep RL is hard to pinpoint, but personally, I recommend the following survey:

Y. Li, “Deep Reinforcement Learning, ” arXiv:1810.06339 [cs, stat], Oct. 2018, Accessed: May 08, 2020. [Online]. Available: http://arxiv.org/abs/1810.06339.

Courses

  • Reinforcement Learning Specialization on Coursera: At the time of this writing, this is a rather new course. It is ministered by Martha and David White from UAlberta and has been endorsed by Richard Sutton himself. Now, I’m not very fond of using an appeal to authority to make a point, so I actually enrolled in the course and started doing it. I believe that, as of now, this is the most educational and informative resource available online to learn the fundamentals of RL from scratch. The professors really know what they’re talking about, they use Sutton’s book as reference material (which is freely available online), guide you to details that no one would know are important when studying RL alone and prepare you to venture further into the area, with a solid foundation. I definitely recommend this as a starting point for anyone who wants to dig deep into RL.

  • Introduction to reinforcement learning by David Silver [slides]: Before the RL specialization, I believe this playlist was the go-to reference to study RL. David Silver is also a reference in the area and also uses Sutton’s book as a reference for the course. My only grudge with the course is that it assumes considerable previous knowledge in many areas, at times sounding more like a graduate course, encouraging students to prove statements while skipping K-armed bandits, for example.

  • Deep RL Bootcamp [slides]: lectures with some of the scientists behind the groundbreaking DRL algorithms created in the last years. Good explanations of DQN and policy gradient methods from their creators.

  • CS 285 at UC Berkeley: a full DRL course. This looks like the real deal. Unfortunately, it was not the material I used to learn what I know, but people highly recommend it, so I’ll keep it in the list.

  • Practical_RL: An open course on reinforcement learning in the wild […] maintained to be friendly to online students (both english and russian). This one is mostly maintained on GitHub. Unfortunately, lectures are in Russian, but there are slides and links to more material.

Miscellaneous online material

  • Spinning Up in Deep RL: an educational resource produced by OpenAI that makes it easier to learn about deep reinforcement learning. The website has great summaries about the policy gradient algorithms created by OpenAI, as well as references to the original papers.

  • Arthur Juliani’s Simple Reinforcement learning series.

Algorithm implementations

  • My personal repository where I have implemented policy iteration, value iteration, Monte Carlo ES, Sarsa and Q-Learning in C++ and applied it in a grid world.

  • MorvanZhou’s simple and educational Python implementations of classical RL algorithms. I used them on a personal project and compiled them into a single, documented Python module available here under the same license.

  • Implementations of model-free and policy gradient methods in PyTorch 0.4, in a very instructive way:

  • OpenAI baselines: Baseline implementations of most famous DRL algorithms by OpenAI (since the ones they found online were riddled with bugs).

  • Stable baselines: a repository that is maintained more frequently than OpenAI’s. Their code is PEP8 compliant and actually documented.

  • Arthur Juliani also has complete implementations over at his GitHub, as well as commented ones in his aforementioned series.

  • Denny Britz has a very famous repository of reinforcement algorithms over at GitHub.

  • PyTorch official DQN tutorial: this is the intermediate step in the PyTorch tutorials so, if you fancy learning some PyTorch, I believe this is the most straightforward way to implement and debug your first DRL algorithm.

Twitter, Facebook