Advanced Reinforcement Learning

Advanced Reinforcement Learning

1.1 Actor-Critic Reinforcement Learning

Reinforcement Learning learns a policy from environmental experiences, it gains reward signals from environment and accordingly adjust the model to maximize the expected rewards and thus formulate a policy. Value-based Reinforcement Learning define value functions that evaluate the states and actions of a markov decision process, the optimal policy is typically selected from greedy policy. Whereas the Policy-based Reinforcement Learning algorithms model the policy explicitly and optimize the policy by methods like gradient ascent.

Another group of algorithm integrates the advantages of both Value-based methods and Policy-based methods, namely Actor-Critic architectures, in Actor-Critic, we define a critic:

$$Q_w(s,a)\approx Q^{{\pi }_{\theta }}(s,a)$$

Where the critic approximates state-action value function $Q(s,a)$, the actor approximates the policy $\pi$, there are parameterized by $w$ and $\theta$ respectively.

Actor-critic algorithm follow an approximate policy gradient, the actor network can be updated by:

$$\begin{gathered} {\mathrm{\nabla }}_{\theta }J(\theta )\approx {\mathbb{E}}_{{\pi }_{\theta }}[{\mathrm{\nabla }}_{\theta }log{\pi }_{\theta }(s,a)Q_w(s,a)] \\ \mathrm{\nabla }\theta =\alpha {\mathrm{\nabla }}_{\theta }log{\pi }_{\theta }(s,a)Q_w(s,a) \\ {\theta }_{t+1}\leftarrow {\theta }_t+\lambda \mathrm{\nabla }{\theta }_t \end{gathered}$$

The critic network is based on critic functions, here use Q-function as an example, the critic network can be updated as:

$$\begin{gathered} {\mathrm{\nabla }}_{w}J(w)\approx {\mathbb{E}}_{{\pi }_{\theta }}[MSE(Q_t,R_{t+1}+ma{x}_a{Q}_{t+1})] \\ \mathrm{\nabla }w=MSE(Q_t,R_{t+1}+ma{x}_a{Q}_{t+1}) \\ {w}_{t+1}\leftarrow w_t+\lambda \mathrm{\nabla }{w}_t \end{gathered}$$

Instead of leting critic to estimate state-value function, we can allow it to alternatively estimates Advantage function $A(s,a)=Q_w(s,a)-V_v(s)$ to reduce the variance. There are many alternative critic function choices.

1.2 Proximal Policy Optimization with AC-based Advantage Function

Baseline algorithm of OpenAI. PPO allows off-policy learning to policy gradient algorithm. In policy gradient, we update our policy network by compute gradient of the expected reward function with respect to policy parameters $\theta$, and perform gradient acsent to maximize it:

$$\begin{gathered} \mathrm{\nabla }J(\theta )=\frac{1}{N}\sum _{n=1}^N\sum _{t=1}^{T_n}R({\tau }^n){\mathrm{\nabla }}_{\theta }log{\pi }_{\theta }(a_t^n|s_t^n) \\ ={\mathbb{E}}_{\tau \backsim {\pi }_{\theta }}[R({\tau }^n){\mathrm{\nabla }}_{\theta }log{\pi }_{\theta }(a_t^n|s_t^n)] \\ \theta \leftarrow \theta +\alpha \mathrm{\nabla }{\overline{R}}_{\theta } \end{gathered}$$

In PPO algorithm, instead of sampling trajectories from policy ${\pi }_{\theta }$, in order to increase sample efficiency(reuse expirence), we sample trajectories from another policy ${\pi }_{{\theta }^{\prime }}$ and apply a importance sampling method to correct the difference.

$$\begin{gathered} J_{{\theta }^{\prime }}(\theta )={\mathbb{E}}_{\tau \backsim {\pi }_{{\theta }^{\prime }}}[\frac{{\pi }_{\theta }(a_t|s_t)}{{\pi }_{{\theta }^{\prime }}(a_t|s_t)}R({\tau }^n)] \\ \mathrm{\nabla }{J}_{{\theta }^{\prime }}(\theta )={\mathbb{E}}_{\tau \backsim {\pi }_{{\theta }^{\prime }}}[\frac{{\pi }_{\theta }(a_t|s_t)}{{\pi }_{{\theta }^{\prime }}(a_t|s_t)}R({\tau }^n){\mathrm{\nabla }}_{\theta }log{\pi }_{\theta }(a_t^n|s_t^n)] \end{gathered}$$

The reward $R({\tau }^n)$ can be replaced by advantage function $A^{{\theta }^{\prime }}(s_t,a_t)$, where we levarage the power of AC architecture the gain more suitable representation of the loss to optimize.

$$\begin{gathered} J_{{\theta }^{\prime }}(\theta )={\mathbb{E}}_{\tau \backsim {\pi }_{{\theta }^{\prime }}}[\frac{{\pi }_{\theta }(a_t|s_t)}{{\pi }_{{\theta }^{\prime }}(a_t|s_t)}{A}^{{\theta }^{\prime }}(s_t,a_t)] \\ \mathrm{\nabla }{J}_{{\theta }^{\prime }}(\theta )={\mathbb{E}}_{\tau \backsim {\pi }_{{\theta }^{\prime }}}[\frac{{\pi }_{\theta }(a_t|s_t)}{{\pi }_{{\theta }^{\prime }}(a_t|s_t)}{A}^{{\theta }^{\prime }}(s_t,a_t){\mathrm{\nabla }}_{\theta }log{\pi }_{\theta }(a_t^n|s_t^n)] \end{gathered}$$

1.2.1 PPO KL Regularzation

In addition, we need to add a regularzation term (or in TRPO, add a constrain) to the objective function to constrain the difference between two distributions, therefore the objective function becomes: $J(\theta )=J_{{\theta }^{\prime }}(\theta )-\beta KL(\theta ,{\theta }^{\prime })$ Where adaptively set the value of $\beta$, specificlly when $KL(\theta ,{\theta }^{\prime })>K{L}_{max}$, increase $\beta$; when $KL(\theta ,{\theta }^{\prime })<K{L}_{min}$, decrease $\beta$.

1.2.1 PPO Clip Method

$$J_{clip}(\theta )\approx \sum _{s_t,a_t}min(\frac{p_{\theta }(a_t|s_t)}{p_{{\theta }^{\prime }}(a_t|s_t)}{A}^{{\theta }^{\prime }}(s_t,a_t),clip(\frac{p_{\theta }(a_t|s_t)}{p_{{\theta }^{\prime }}(a_t|s_t)},1-ϵ,1+ϵ)A^{{\theta }^{\prime }}(s_t,a_t))$$

Where $clip$ = $1-ϵ$ if $\frac{p_{\theta }(a_{t}given{s}_t)}{p_{{\theta }^{\prime }}(a_{t}given{s}_t)}<1-ϵ$ ; $clip$ = $1+ϵ$ if $\frac{p_{\theta }(a_{t}given{s}_t)}{p_{{\theta }^{\prime }}(a_{t}given{s}_t)}>1+ϵ$ ; values that fall anywhere in between $clip=\frac{p_{\theta }(a_{t}given{s}_t)}{p_{{\theta }^{\prime }}(a_{t}given{s}_t)}$. This methods hence set an constrain that making sure the differences between two policy distributions changes within certain range of $ϵ$.