强化学习的一些数学基础

date: 2025-10-09
tags: 数学

Performance Difference Lemma

这个引理非常厉害，它相当于是可以用老策略的 adv 来估计新策略的提升量。我们就可以就此得到一些提升的下界。

\begin{aligned} V^\pi(s_0)&=\mathbb{E}_{a_0\sim\pi(\cdot|s_0)}[r(s_0,a_0)+\gamma\mathbb{E}_{s'\sim P(s_0,a_0)}V^\pi(s')]\\ Q^\pi(s_0,a_0)&=r(s_0,a_0)+\gamma\mathbb{E}_{s'\sim P(s_0,a_0)}V^\pi(s') \end{aligned}

定义： $\mathbb{P}_h^\pi(s;s_0)$ 是在 $\pi$ 下，通过 $h$ 步从 $s_0$ 转换到 $s$ 的概率。 $\mathbb{P}_h^\pi(s,a;s_0)$ 是在 $\pi$ 下，通过 $h$ 步从 $s_0$ 转换到 $s$ ，然后在 $s$ 选择了 $a$ 动作的概率。也就有：

\mathbb{P}_h^\pi(s;s_0)=\sum_a\mathbb{P}^\pi_h(s,a;s_0)

定义：

d_{s_0}^\pi(s,a)=(1-\gamma)\sum_h^\infty\gamma^h\mathbb{P}_h^\pi(s,a;s_0)

引理：Performance Difference Lemma

V^\pi(s_0) -V^{\pi'}(s_0)=\frac{1}{1-\gamma}\mathbb{E}_{s\sim d_{s_0}^\pi}[\mathbb{E}_{a\sim\pi(\cdot|s)}A^{\pi'}(s,a)]

证明：

\begin{aligned} &V^\pi(s_0)-V^{\pi'}(s_0)\\ &=V^\pi(s_0) -\mathbb{E}_{a_0\sim\pi(\cdot|s_0)}[Q^{\pi'}(s_0,a_0)]+\mathbb{E}_{a_0\sim\pi(\cdot|s_0)}[Q^{\pi'}(s_0,a_0)]-V^{\pi'}(s_0)\\ &= V^\pi(s_0) -\mathbb{E}_{a_0\sim\pi(\cdot|s_0)}[r(s_0,a_0)+\gamma\mathbb{E}_{s'\sim P(s_0,a_0)}V^{\pi'}(s')]+\mathbb{E}_{a_0\sim\pi(\cdot|s_0)}[Q^{\pi'}(s_0,a_0)]-V^{\pi'}(s_0)\\ &=\mathbb{E}_{a_0\sim\pi(\cdot|s_0)}[r(s_0,a_0)+\gamma\mathbb{E}_{s'\sim P(s_0,a_0)}V^\pi(s')]-\mathbb{E}_{a_0\sim\pi(\cdot|s_0)}[r(s_0,a_0)+\gamma\mathbb{E}_{s'\sim P(s_0,a_0)}V^{\pi'}(s')]\\ &+\mathbb{E}_{a_0\sim\pi(\cdot|s_0)}[Q^{\pi'}(s_0,a_0)]-V^{\pi'}(s_0)\\ &=\gamma\mathbb{E}_{a_0\sim\pi(\cdot|s_0)}[\mathbb{E}_{s_1\sim P(s_0,a_0)}[V^\pi(s_1)-V^{\pi'}(s_1)]]+\mathbb{E}_{a_0\sim\pi(\cdot|s_0)}[Q^{\pi'}(s_0,a_0)]-V^{\pi'}(s_0)\\ &=\gamma\mathbb{E}_{a_0\sim\pi(\cdot|s_0)}[\mathbb{E}_{s_1\sim P(s_0,a_0)}[V^\pi(s_1)-V^{\pi'}(s_1)]]+\mathbb{E}_{a_0\sim\pi(\cdot|s_0)}[Q^{\pi'}(s_0,a_0)-V^{\pi'}(s_0)]\\ &=\gamma\mathbb{E}_{a_0\sim\pi(\cdot|s_0)}[\mathbb{E}_{s_1\sim P(s_0,a_0)}[V^\pi(s_1)-V^{\pi'}(s_1)]]+\mathbb{E}_{a_0\sim\pi(\cdot|s_0)}[A^{\pi'}(s_0,a_0)]\\ \end{aligned}

如果我们设：

\mathbb{P}^\pi(s_1;s_0)=\sum_{a_0}\pi(a_0|s_0)P(s_1|s_0,a_0)=\mathbb{E}_{a_0\sim\pi(\cdot|s_0)}P(s_1|s_0,a_0)

我们也就有了：

\begin{aligned} V^\pi(s_0)-V^{\pi'}(s_0)=\gamma\mathbb{E}_{s_1\sim\mathbb{P}^{\pi}(\cdot|s_0)}[V^\pi(s_1)-V^{\pi'}(s_1)]+\mathbb{E}_{a_0\sim\pi(\cdot|s_0)}[A^{\pi'}(s_0,a_0)]\\ \end{aligned}

替换一下，我们有：

\begin{aligned} V^\pi(s_1)-V^{\pi'}(s_1)=\gamma\mathbb{E}_{s_2\sim\mathbb{P}^{\pi}(\cdot|s_1)}[V^\pi(s_2)-V^{\pi'}(s_2)]+\mathbb{E}_{a_1\sim\pi(\cdot|s_1)}[A^{\pi'}(s_1,a_1)]\\ \end{aligned}

代入就有：

\begin{aligned} V^\pi(s_0)-V^{\pi'}(s_0)&= \gamma\mathbb{E}_{s_1\sim\mathbb{P}_1^{\pi}(\cdot|s_0)}[\gamma\mathbb{E}_{s_2\sim\mathbb{P}_1^{\pi}(\cdot|s_1)}[V^\pi(s_2)-V^{\pi'}(s_2)] +\mathbb{E}_{a_1\sim\pi(\cdot|s_1)}[A^{\pi'}(s_1,a_1)]]\\ &+\mathbb{E}_{a_0\sim\pi(\cdot|s_0)}[A^{\pi'}(s_0,a_0)]\\ &=\gamma^2\mathbb{E}_{s_2\sim\mathbb{P}_2^{\pi}(\cdot|s_0)}[V^\pi(s_2)-V^{\pi'}(s_2)]+\gamma\mathbb{E}_{s_1\sim\mathbb{P}_1^{\pi}(\cdot|s_0)}[\mathbb{E}_{a_1\sim\pi(\cdot|s_1)}[A^{\pi'}(s_1,a_1)]]\\ &+\mathbb{E}_{a_0\sim\pi(\cdot|s_0)}[A^{\pi'}(s_0,a_0)]\\ &=\gamma^2\mathbb{E}_{s_2\sim\mathbb{P}_2^{\pi}(\cdot|s_0)}[V^\pi(s_2)-V^{\pi'}(s_2)]+\gamma\mathbb{E}_{s_1,a_1\sim\mathbb{P}_1^{\pi}(\cdot,\cdot|s_0)}[A^{\pi'}(s_1,a_1)]\\ &+\mathbb{E}_{a_0\sim\pi(\cdot|s_0)}[A^{\pi'}(s_0,a_0)]\\ &=...\\ &=\sum_{h=0}^{\infty}\gamma^h\mathbb{E}_{s,a\sim \mathbb{P}_h^\pi(\cdot,\cdot;s_0)}A^{\pi'}(s,a)\\ &=\frac{1}{1-\gamma}\mathbb{E}_{s,a\sim d_{s_0}^\pi}A^{\pi'}(s,a) \end{aligned}