The Policy Gradient (PG) method is a popular policy-based approach in reinforcement learning (RL) that directly optimizes the policy function by changing its parameters using gradient ascent. In PG, the policy π is represented by a parametric function, such as a neural network, that maps state to actions. The goal of PG is to obtain the maximum cumulative reward over a trajectory of interactions, and to do so, the Stochastic Gradient Ascent algorithm is used to optimize the utility function. However, PG can suffer from issues such as high variance in the gradient estimates, which can lead to slower convergence and/or instability. In this article, PG is explained in detail, and an implementation of the algorithm is provided using PyTorch. Multiple modifications and extensions proposed to improve PG’s performance are also discussed.

source update: Policy Gradient Algorithm’s Mathematics Explained with PyTorch… – Towards AI