RL-Adventure-2: Policy Gradients

PyTorch tutorial of: actor critic / proximal policy optimization / acer / ddpg / twin dueling ddpg / soft actor critic / generative adversarial imitation learning / hindsight experience replay

The deep reinforcement learning community has made several improvements to the policy gradient algorithms. This tutorial presents latest extensions in the following order:

1. Advantage Actor Critic (A2C)

• actor-critic.ipynb

• A3C Paper

• OpenAI blog

2. High-Dimensional Continuous Control Using Generalized Advantage Estimation

• gae.ipynb

• GAE Paper

3. Proximal Policy Optimization Algorithms

• ppo.ipynb

• PPO Paper

• OpenAI blog

4. Sample Efficient Actor-Critic with Experience Replay

• acer.ipynb

• ACER Paper

5. Continuous control with deep reinforcement learning

• ddpg.ipynb

• DDPG Paper

6. Addressing Function Approximation Error in Actor-Critic Methods

• td3.ipynb

• Twin Dueling DDPG Paper

7. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor

• soft actor-critic.ipynb

• Soft Actor-Critic Paper

8. Generative Adversarial Imitation Learning

• gail.ipynb

• GAIL Paper

9. Hindsight Experience Replay

• her.ipynb

• HER Paper

• OpenAI Blog

If you get stuck…

• Remember you are not stuck unless you have spent more than a week on a single algorithm. It is perfectly normal if you do not have all the required knowledge of mathematics and CS.

• Carefully go through the paper. Try to see what is the problem the authors are solving. Understand a high-level idea of the approach, then read the code (skipping the proofs), and after go over the mathematical details and proofs.

RL Algorithms

Deep Q Learning tutorial: DQN Adventure: from Zero to State of the Art  Awesome RL libs: rlkit @vitchyr, pytorch-a2c-ppo-acktr @ikostrikov, ACER @Kaixhin

Best RL courses

• Berkeley deep RL link

• Deep RL Bootcamp link

• David Silver's course link

• Practical RL link