Free online resources to study reinforcement learning

Over the last few years and months, I gathered some material that I used to study reinforcement learning and deep reinforcement learning, mostly by myself, as part of my PhD formation.

I hope these links can help others as much as they’ve helped me. The list also serves the purpose of remembering me that this material exists, since memory loss is a thing during a PhD.

Books and surveys

  • R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed. Cambridge, Mass: The MIT Press, 2018. Freely and legally available here (you can find my name in the errata, lol)

  • Y. Li, “Deep Reinforcement Learning,” arXiv:1810.06339 [cs, stat], Oct. 2018, Accessed: May 08, 2020. [Online]. Available: http://arxiv.org/abs/1810.06339.

  • P. Hernandez-Leal, B. Kartal, and M. E. Taylor, “A Survey and Critique of Multiagent Deep Reinforcement Learning,” Auton Agent Multi-Agent Syst, vol. 33, no. 6, pp. 750–797, Nov. 2019, doi: 10.1007/s10458-019-09421-1. Available: http://arxiv.org/abs/1810.05587.

Courses

  • Introduction to reinforcement learning by David Silver [slides]: I highly recommend pairing this one up with Sutton’s book, as the lessons are heaviliy influenced by it.

  • Deep RL Bootcamp [slides]: lectures with some of the scientists behind the groundbreaking DRL algorithms created in the last years. Good explanations of DQN and policy gradient methods from their creators.

  • CS 285 at UC Berkeley: a full DRL course. This looks like the real deal. Unfortunately, it was not the material I used to learn what I know, but people highly recommend it, so I’ll keep it in the list.

  • Practical_RL: An open course on reinforcement learning in the wild […] maintained to be friendly to online students (both english and russian). This one is mostly maintained on GitHub. Unfortunately, lectures are in Russian, but there are slides and links to more material.

Miscellaneous online material

  • Spinning Up in Deep RL: an educational resource produced by OpenAI that makes it easier to learn about deep reinforcement learning. The website has great summaries about the policy gradient algorithms created by OpenAI, as well as references to the original papers.

  • Arthur Juliani’s Simple Reinforcement learning series

Algorithm implementations

  • My personal repository where I have implemented policy iteration, value iteration, Monte Carlo ES, Sarsa and Q-Learning in C++ and applied it in a grid world.

  • MorvanZhou’s simple and educational Python implementations of classical RL algorithms. I used them on a personal project and compiled them into a single, documented Python module available here under the same license.

  • Implementations of model-free and policy gradient methods in PyTorch 0.4, in a very instructive way:

  • OpenAI baselines: Baseline implementations of most famous DRL algorithms by OpenAI (since the ones they found online were riddled with bugs).

  • Stable baselines: a repository that is maintained more frequently than OpenAI’s. Their code is PEP8 compliant and actually documented.

  • Arthur Juliani also has complete implementations over at his GitHub, as well as commented ones in his aforementioned series.

  • Denny Britz has a very famous repository of reinforcement algorithms over at GitHub.

  • PyTorch official DQN tutorial: this is the intermediate step in the PyTorch tutorials so, if you fancy learning some PyTorch, I believe this is the most straightforward way to implement and debug your first DRL algorithm.

Written on May 10, 2020