1<!DOCTYPE html> 2<html lang="en"><head> 3 <meta charset="utf-8"> 4 <meta http-equiv="X-UA-Compatible" content="IE=edge"> 5 <meta name="viewport" content="width=device-width, initial-scale=1"> 6
6<script src="https://cdn.jsdelivr.net/npm/chart.js"></script>
6<!-- Begin Jekyll SEO tag v2.8.0 --> 7<title>Post-Training Primer | No-Range Space</title> 8<meta name="generator" content="Jekyll v3.10.0" /> 9<meta property="og:title" content="Post-Training Primer" /> 10<meta name="author" content="Nikita Durasov" /> 11<meta property="og:locale" content="en_US" /> 12<meta name="description" content="A first-principles look at post-training for language models, tracing how policy gradients lead to PPO and GRPO. The post breaks down reward-to-go, advantage estimation, probability-ratio clipping, and group-relative rewards, while highlighting the practical differences between PPOâs learned critic and GRPOâs critic-free approach. It also covers implementation details and common pitfalls when putting these methods into practice." /> 13<meta property="og:description" content="A first-principles look at post-training for language models, tracing how policy gradients lead to PPO and GRPO. The post breaks down reward-to-go, advantage estimation, probability-ratio clipping, and group-relative rewards, while highlighting the practical differences between PPOâs learned critic and GRPOâs critic-free approach. It also covers implementation details and common pitfalls when putting these methods into practice." /> 14<link rel="canonical" href="https://www.norange.ai/post-training/2026/08/24/post-training-primer.html" /> 15<meta property="og:url" content="https://www.norange.ai/post-training/2026/08/24/post-training-primer.html" /> 16<meta property="og:site_name" content="No-Range Space" /> 17<meta property="og:type" content="article" /> 18<meta property="article:published_time" content="2026-08-24T00:00:00+00:00" /> 19<meta name="twitter:card" content="summary" /> 20<meta property="twitter:title" content="Post-Training Primer" />
21<script type="application/ld+json"> 22{"@context":"https://schema.org","@type":"BlogPosting","author":{"@type":"Person","name":"Nikita Durasov"},"dateModified":"2026-08-24T00:00:00+00:00","datePublished":"2026-08-24T00:00:00+00:00","description":"A first-principles look at post-training for language models, tracing how policy gradients lead to PPO and GRPO. The post breaks down reward-to-go, advantage estimation, probability-ratio clipping, and group-relative rewards, while highlighting the practical differences between PPOâs learned critic and GRPOâs critic-free approach. It also covers implementation details and common pitfalls when putting these methods into practice.","headline":"Post-Training Primer","mainEntityOfPage":{"@type":"WebPage","@id":"https://www.norange.ai/post-training/2026/08/24/post-training-primer.html"},"url":"https://www.norange.ai/post-training/2026/08/24/post-training-primer.html"}</script>
22 23<!-- End Jekyll SEO tag --> 24<link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro" 25 rel="stylesheet"> 26 <link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css" integrity="sha384-ggOyR0iXCbMQv3Xipma34MD+dH/1fQ784/j6cY/iJTQUOhcWr7x9JvoRxT2MZw1T" crossorigin="anonymous"> 27 <link rel="stylesheet" href="/assets/css/style.css"> 28 <link rel="shortcut icon" type="image/png" href="/assets/images/logo.png"> 29 <link href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.0.0/css/all.min.css" rel="stylesheet"> 30 <!--
30<script async src="https://busuanzi.icodeq.com/busuanzi.pure.mini.js"></script>
30 --> 31 <!--
31<script defer src="https://busuanzi.9420.ltd/js"></script>
31 --> 32 <!-- <meta name="referrer" content="no-referrer-when-downgrade"> --> 33 34 <meta property="og:image" content="/assets/images/logo.png" /><link type="application/atom+xml" rel="alternate" href="https://www.norange.ai/feed.xml" title="No-Range Space" />
vendor: 63 bytes, line 34
34<script async src="https://www.googletagmanager.com/gtag/js?id=
34UA-142853940-1
vendor: 12 bytes, line 34
34"></script>
35<script> 36 window['ga-disable-UA-142853940-1'] = window.doNotTrack === "1" || navigator.doNotTrack === "1" || navigator.doNotTrack === "yes" || navigator.msDoNotTrack === "1"; 37
vendor: 134 bytes, lines 37-41
37window.dataLayer = window.dataLayer || []; 38 function gtag(){dataLayer.push(arguments);} 39 gtag('js', new Date()); 40 41 gtag('config', '
41UA-142853940-1
vendor: 4 bytes, line 41
41');
42</script>
42 43<style> 44 .MathJax_Display, .MJXc-display, .MathJax_SVG_Display { 45 overflow-x: auto; 46 overflow-y: hidden; 47 } 48 </style> 49 <!-- 50<meta property="og:image" content="https://www.norange.ai/assets/logo.png" />--> 51</head> 52<body style="font-family: 'Lucida Console', Monaco, monospace"><header class="site-header"> 53 54 <div class="wrapper"><div class="row"> 55 <div class="col-6 col-sm-3"> 56 57 <a rel="author" href="/" class="blog-btn">Blog â</a> 58 59 <style> 60 .blog-btn { 61 display: inline-block; 62 margin: 2px 0; 63 line-height: 25px; /* space above & below the button */ 64 padding: 3px 10px; /* space inside the button around the text */ 65 font-family: 'Inter', sans-serif; 66 font-size: 18px; 67 font-weight: 500; 68 color: #fff; 69 text-decoration: none; 70 border-radius: 8px; 71 border: 1.5px solid transparent; 72 background: linear-gradient(#000, #000) padding-box, 73 linear-gradient(270deg, #096ce2, #5d41dc) border-box; 74 transition: all 0.4s ease; 75 box-shadow: 0 0 12px rgba(131, 56, 236, 0.6); 76 } 77 78 .blog-btn:hover { 79 background: linear-gradient(#000, #000) padding-box, 80 linear-gradient(90deg, #FFFFFF, #f285c1) border-box; 81 box-shadow: 0 0 18px #f285c1; 82 transform: translateY(-1px); 83 } 84 85 .blog-btn:hover::after { 86 position: relative; 87 left: 3px; 88 transition: left 0.0s ease; 89 } 90 91 </style> 92 93 </div> 94 95 <div class="col-6 col-sm-9"><nav class="site-nav" style="z-index: 1;"> 96 <input type="checkbox" id="nav-trigger" class="nav-trigger" /> 97 <label for="nav-trigger"> 98 <span class="menu-icon"> 99 <svg viewBox="0 0 18 15" width="18px" height="15px"> 100 <path d="M18,1.484c0,0.82-0.665,1.484-1.484,1.484H1.484C0.665,2.969,0,2.304,0,1.484l0,0C0,0.665,0.665,0,1.484,0 h15.032C17.335,0,18,0.665,18,1.484L18,1.484z M18,7.516C18,8.335,17.335,9,16.516,9H1.484C0.665,9,0,8.335,0,7.516l0,0 c0-0.82,0.665-1.484,1.484-1.484h15.032C17.335,6.031,18,6.696,18,7.516L18,7.516z M18,13.516C18,14.335,17.335,15,16.516,15H1.484 C0.665,15,0,14.335,0,13.516l0,0c0-0.82,0.665-1.483,1.484-1.483h15.032C17.335,12.031,18,12.695,18,13.516L18,13.516z"/> 101 </svg> 102 </span> 103 </label> 104 105 <div class="trigger" style="z-index: 1;"><a class="page-link-local" href="/about/">about</a></div> 106 </nav></div> 107 </div> 108 109 <!-- <a class="site-title" rel="author" href="/">No-Range Space</a><nav class="site-nav" style="z-index: 1;"> 110 <input type="checkbox" id="nav-trigger" class="nav-trigger" /> 111 <label for="nav-trigger"> 112 <span class="menu-icon"> 113 <svg viewBox="0 0 18 15" width="18px" height="15px"> 114 <path d="M18,1.484c0,0.82-0.665,1.484-1.484,1.484H1.484C0.665,2.969,0,2.304,0,1.484l0,0C0,0.665,0.665,0,1.484,0 h15.032C17.335,0,18,0.665,18,1.484L18,1.484z M18,7.516C18,8.335,17.335,9,16.516,9H1.484C0.665,9,0,8.335,0,7.516l0,0 c0-0.82,0.665-1.484,1.484-1.484h15.032C17.335,6.031,18,6.696,18,7.516L18,7.516z M18,13.516C18,14.335,17.335,15,16.516,15H1.484 C0.665,15,0,14.335,0,13.516l0,0c0-0.82,0.665-1.483,1.484-1.483h15.032C17.335,12.031,18,12.695,18,13.516L18,13.516z"/> 115 </svg> 116 </span> 117 </label> 118 119 <div class="trigger" style="z-index: 1;"><a class="page-link-local" href="/about/">about</a></div> 120 </nav>--> 121 122 </div> 123</header> 124<main class="page-content" aria-label="Content"> 125 <div class="wrapper"> 126 <div style="position: relative; z-index: 0;"> 127
127<script type="text/javascript" 128 src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS_CHTML"></script>
vendor: 1 bytes, line 128
128
129<script src="https://cdn.jsdelivr.net/npm/chart.js"></script>
vendor: 1 bytes, line 129
129
130<script src="https://cdn.jsdelivr.net/npm/[email protected]"></script>
130 131 132<nav class="post-toc" aria-label="Table of contents"> 133 <p class="post-toc-title">Contents</p> 134</nav> 135 136<article class="post h-entry" itemscope itemtype="http://schema.org/BlogPosting"> 137 138 <header class="post-header"> 139 <h1 class="post-title p-name" itemprop="name headline"><span class="post-title-gradient">Post-Training Primer</span></h1> 140 <p class="post-meta" style="margin: 10px 0px"> 141 <span class="title-highlights" style="font-size:15px"><time class="dt-published" datetime="2026-08-24T00:00:00+00:00" itemprop="datePublished"> 142 Aug 24, 2026 143 </time>⢠144 <span itemprop="author" itemscope itemtype="http://schema.org/Person"> 145 <span class="p-author h-card" itemprop="name">Nikita Durasov</span></span><span style="margin-left: 8px;">â¢</span> <i class="fa-solid fa-eye"></i> 146 <span><span id="ga_site_pv">Loading...</span> views</span> 147 </span> 148 </p> 149 </header> 150 151 <div class="post-content e-content" itemprop="articleBody"> 152 <style> 153.reference { 154 color: #7c3aed; 155 font-weight: 650; 156} 157.readable-code pre, 158.readable-code code { 159 color: #e5e7eb !important; 160 background: #111827 !important; 161} 162.readable-code pre { 163 border: 1px solid #374151; 164 border-radius: 10px; 165 padding: 1rem 1.1rem; 166} 167</style> 168 169<h2 id="introduction">Introduction</h2> 170 171<h3 id="post-training-unified">Post-Training Unified</h3> 172 173<p>Pretraining teaches a model to represent and continue the distribution of its training data. Post-training changes that distribution toward behavior we care about: following instructions, solving tasks, respecting preferences, using tools, or maximizing a measurable reward.</p> 174 175<p>For language models, the object being trained is still a conditional probability distribution,</p> 176 177\[\pi_{\theta}(o_t \mid q, o_{<t}),\] 178 179<p>where \(q\) is a prompt, \(o_{<t}\) is the response prefix, and \(o_t\) is the next token. What changes between supervised fine-tuning, PPO, and GRPO is primarily the coefficient placed in front of the familiar log-likelihood gradient</p> 180 181\[\nabla_{\theta}\log \pi_{\theta}(o_t \mid q,o_{<t}).\] 182 183<p>This gives a useful unifying picture:</p> 184 185<ul> 186 <li> 187 <p>Supervised fine-tuning increases the probability of selected demonstrations.</p> 188 </li> 189 <li> 190 <p>Policy gradients increase or decrease the probability of sampled actions according to their return.</p> 191 </li> 192 <li> 193 <p>PPO uses an advantage estimate and constrains how far the new policy can move from the policy that generated the data.</p> 194 </li> 195 <li> 196 <p>GRPO retains the PPO update but estimates the baseline by comparing several responses to the same prompt instead of training a critic.</p> 197 </li> 198</ul> 199 200<p>The differences between these methods then come down to a more subtle question: what signal should multiply the log-likelihood gradient, and why? That question leads naturally from likelihood to return, from return to advantage, and from advantage to the particular updates used by PPO and GRPO.</p> 201 202<h3 id="reinforcement-learning-notation">Reinforcement Learning Notation</h3> 203 204<p>Consider an episodic Markov decision process. At step \(t\), an agent observes a state \(s_t\), samples an action \(a_t\), receives a reward \(r_t\), and moves to \(s_{t+1}\). A trajectory is</p> 205 206\[\tau=(s_0,a_0,r_0,s_1,a_1,r_1,\ldots,s_T).\] 207 208<p>The policy is a probability distribution</p> 209 210\[\pi_{\theta}(a_t\mid s_t),\] 211 212<p>parameterized by \(\theta\). For now, define the undiscounted trajectory return as</p> 213 214\[R(\tau)=\sum_{t=0}^{T-1}r_t.\] 215 216<p>The training objective is</p> 217 218\[J(\theta)=\mathbb{E}_{\tau\sim\pi_{\theta}}[R(\tau)].\] 219 220<p>For a language model, the translation is direct:</p> 221 222\[s_t=(q,o_{<t}),\qquad a_t=o_t.\] 223 224<p>The environment state is the prompt and generated prefix, and the action is the next token. The probability of a complete response factorizes autoregressively:</p> 225 226\[\pi_{\theta}(o\mid q) 227 228= 229 230\prod_{t=1}^{|o|}\pi_{\theta}(o_t\mid q,o_{<t}).\] 231 232<h2 id="policy-gradient">Policy Gradient</h2> 233 234<h3 id="reinforce">REINFORCE</h3> 235 236<p>The probability of a trajectory is</p> 237 238\[p_{\theta}(\tau) 239 240= 241 242p(s_0) 243 244\prod_{t=0}^{T-1} 245 246\pi_{\theta}(a_t\mid s_t) 247 248P(s_{t+1}\mid s_t,a_t).\] 249 250<p>The initial-state distribution and environment dynamics do not depend on the policy parameters. Starting from the objective,</p> 251 252\[J(\theta)=\sum_{\tau}p_{\theta}(\tau)R(\tau),\] 253 254<p>we differentiate:</p> 255 256\[\nabla_{\theta}J(\theta) 257 258= 259 260\sum_{\tau}\nabla_{\theta}p_{\theta}(\tau)R(\tau).\] 261 262<p>Use the log-derivative identity</p> 263 264<details class="margin-note"> 265 <summary>What is log-derivative trick?</summary> 266 267 Start with the derivative of the logarithm: 268 269 $$ 270 271 \nabla_{\theta}\log p_{\theta}(\tau) 272 = 273 \frac{1}{p_{\theta}(\tau)} 274 \nabla_{\theta}p_{\theta}(\tau) 275 276 $$ 277 278 Multiplying both sides by \(p_{\theta}(\tau)\) gives: 279 280 $$ 281 282 \nabla_{\theta}p_{\theta}(\tau) 283 = 284 p_{\theta}(\tau) 285 \nabla_{\theta}\log p_{\theta}(\tau) 286 287 $$ 288 289 This is the <strong>log-derivative trick</strong>, also called the <strong>score-function identity</strong>. 290</details> 291 292\[\nabla_{\theta}p_{\theta}(\tau) 293 294= 295 296p_{\theta}(\tau)\nabla_{\theta}\log p_{\theta}(\tau).\] 297 298<p>Then</p> 299 300\[\nabla_{\theta}J(\theta) 301 302= 303 304\mathbb{E}_{\tau\sim\pi_{\theta}} 305 306\left[ 307 308R(\tau)\nabla_{\theta}\log p_{\theta}(\tau) 309 310\right].\] 311 312<p>Because only the policy factors depend on \(\theta\),</p> 313 314\[\nabla_{\theta}\log p_{\theta}(\tau) 315 316= 317 318\sum_{t=0}^{T-1} 319 320\color{#2563eb}{\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t)}.\] 321 322<p>Therefore,</p> 323 324\[\nabla_{\theta}J(\theta) 325 326= 327 328\mathbb{E} 329 330\left[ 331 332R(\tau) 333 334\sum_{t=0}^{T-1} 335 336\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t) 337 338\right].\] 339 340<p>This is the score-function or REINFORCE estimator <span class="reference">[Wil92]</span>. It is the central mechanism behind the methods discussed below. We do not differentiate through the sampled discrete action. Instead, we differentiate the log-probability of having sampled it.</p> 341 342<h3 id="why-past-rewards-disappear">Why Past Rewards Disappear</h3> 343 344<p>The previous formula multiplies every policy-score term by the entire trajectory return. At first this seems to imply that an action at time \(10\) should be credited for a reward received at time \(0\). Causally, that cannot be right: the later action cannot change an event that has already happened.</p> 345 346<p>The mathematical statement is exactly</p> 347 348\[\mathbb{E} 349 350\left[ 351 352 r_0\nabla_{\theta}\log\pi_{\theta}(a_{10}\mid s_{10}) 353 354\right] 355 356=0.\] 357 358<p>More generally, for every \(k\lt t\),</p> 359 360\[\mathbb{E} 361 362\left[ 363 364 r_k\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t) 365 366\right] 367 368=0.\] 369 370<h3 id="conditional-expectation-proof">Conditional-expectation proof</h3> 371 372<p>Let \(H_t\) denote everything known immediately before sampling \(a_t\):</p> 373 374\[H_t=(s_0,a_0,r_0,\ldots,s_t).\] 375 376<p>For \(k\lt t\), the reward \(r_k\) is already contained in \(H_t\). It is therefore fixed after conditioning on \(H_t\). By the law of total expectation,</p> 377 378\[\begin{aligned} 379 380&\mathbb{E} 381 382\left[ 383 384 r_k\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t) 385 386\right] 387 388\\\\ 389 390&=\mathbb{E} 391 392\left[ 393 394\mathbb{E} 395 396\left[ 397 398 r_k\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t) 399 400\mid H_t 401 402\right] 403 404\right] 405 406\\\\ 407 408&=\mathbb{E} 409 410\left[ 411 412 r_k 413 414\mathbb{E} 415 416\left[ 417 418\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t) 419 420\mid H_t 421 422\right] 423 424\right]. 425 426\end{aligned}\] 427 428<p>Conditional on the history, the only remaining randomness in this expression is the newly sampled action</p> 429 430\[a_t\sim\pi_{\theta}(\cdot\mid s_t).\] 431 432<p>The expected score of a normalized probability distribution is zero:</p> 433 434\[\begin{aligned} 435 436\mathbb{E}_{a_t\sim\pi_{\theta}} 437 438\left[ 439 440\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t) 441 442\right] 443 444&= 445 446\sum_a 447 448\pi_{\theta}(a\mid s_t) 449 450\nabla_{\theta}\log\pi_{\theta}(a\mid s_t) 451 452\\\\ 453 454&= 455 456\sum_a\nabla_{\theta}\pi_{\theta}(a\mid s_t) 457 458\\\\ 459 460&= 461 462\nabla_{\theta}\sum_a\pi_{\theta}(a\mid s_t) 463 464\\\\ 465 466&= 467 468\nabla_{\theta}1 469 470=0. 471 472\end{aligned}\] 473 474<p>Substituting this into the conditional expectation proves the claim.</p> 475 476<p>A subtle point is worth emphasizing. The past reward \(r_k\) can be highly correlated with the later state \(s_t\). For example, both may depend on earlier actions. Independence is not required. The argument works because, after conditioning on the history, the expected score of the newly sampled action is zero.</p> 477 478<p>The statement concerns</p> 479 480\[\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t),\] 481 482<p>not the gradient of a joint marginal such as \(\nabla_{\theta}\log p_{\theta}(a_t,s_t)\). The distribution of \(s_t\) depends on previous policy decisions, so the latter contains additional terms.</p> 483 484<h3 id="from-the-proof-to-reward-to-go">From the proof to reward-to-go</h3> 485 486<p>Write</p> 487 488\[g_t=\color{#2563eb}{\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t)}.\] 489 490<p>Then</p> 491 492\[\begin{aligned} 493 494\nabla_{\theta}J 495 496&= 497 498\sum_t 499 500\mathbb{E} 501 502\left[ 503 504\left(\sum_k r_k\right)g_t 505 506\right] 507 508\\\\ 509 510&= 511 512\sum_t\sum_k\mathbb{E}[r_k g_t]. 513 514\end{aligned}\] 515 516<p>Every term with \(k\lt t\) is zero. Consequently,</p> 517 518\[\begin{aligned} 519 520\nabla_{\theta}J 521 522&= 523 524\sum_t 525 526\mathbb{E} 527 528\left[ 529 530\left(\sum_{k=t}^{T-1}
530r_k\right)g_t 531 532\right] 533 534\\\\ 535 536&= 537 538\mathbb{E} 539 540\left[ 541 542\sum_t G_t 543 544\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t) 545 546\right], 547 548\end{aligned}\] 549 550<p>where</p> 551 552\[\boxed{ 553 554\color{#16a34a}{G_t}=\sum_{k=t}^{T-1}r_k 555 556}\] 557 558<p>is the reward-to-go for the undiscounted objective. If the objective is</p> 559 560\[J_{\gamma}(\theta) 561 562= 563 564\mathbb{E} 565 566\left[ 567 568\sum_{k=0}^{T-1}\gamma^k r_k 569 570\right],\] 571 572<p>define the locally discounted reward-to-go</p> 573 574<details class="margin-note"><summary>Why introduce discounting?</summary>Discounting makes future rewards count less than immediate rewards. This is useful when we care more about near-term outcomes, when delayed rewards are less certain, or when we want to limit the effective planning horizon. 575<br /><br /> 576It can also reduce the variance of policy-gradient estimates by downweighting distant, noisy rewards, although this comes at the cost of changing the objective: the discounted and undiscounted problems are not equivalent.</details> 577 578\[G_t^{(\gamma)} 579 580= 581 582\sum_{\ell=0}^{T-t-1}\gamma^{\ell}r_{t+\ell}.\] 583 584<p>The exact episodic gradient then contains the discount accumulated before time \(t\):</p> 585 586\[\nabla_{\theta}J_{\gamma} 587 588= 589 590\mathbb{E} 591 592\left[ 593 594\sum_t 595 596\gamma^t G_t^{(\gamma)} 597 598\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t) 599 600\right].\] 601 602<p>Some presentations absorb this outer factor into the discounted state-visitation distribution, so it may not appear explicitly. The convention should be stated rather than silently mixed.</p> 603 604<p>Reward-to-go is therefore not merely a heuristic. It is obtained by deleting terms whose expectation is exactly zero. This normally reduces variance because the score for \(a_t\) is no longer multiplied by irrelevant random rewards from the past.</p> 605 606<h2 id="baselines-and-advantages">Baselines and Advantages</h2> 607 608<p>Throughout this section, expectations are taken over trajectories generated by the current policy \(\pi_\theta\) and the environment dynamics. When we condition on \((s_t)\) or \((s_t,a_t)\), the expectation is instead over the corresponding conditional distribution of future trajectory outcomes. In particular, (G_t) is a sampled return from a trajectory, while (V^\pi) and (Q^\pi) are conditional expectations of that return.</p> 609 610<p>The same zero-score identity lets us subtract a baseline without changing the expected policy gradient. Let (b(H_t)) be any quantity known before sampling (a_t). Then</p> 611 612\[\underset{ 613\substack{ 614s_t\sim d^\pi_t\\ 615a_t\sim\pi_\theta(\cdot\mid s_t) 616} 617}{ 618\mathbb{E} 619} 620\left[ 621b(H_t) 622\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t) 623\right] 624 625=0.\] 626 627<p>Thus,</p> 628 629\[\boxed{ 630 631\nabla_{\theta}J 632 633= 634 635\mathbb{E}_{\tau\sim p_\theta} 636 637\left[ 638 639\sum_t 640 641\left(G_t-b(H_t)\right) 642 643\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t) 644 645\right]. 646 647}\] 648 649<p>The baseline does not need to be constant. It can depend on the state or the full pre-action history. It must not depend on the current sampled action in a way that invalidates the zero-score argument.</p> 650 651<p>If a learned baseline shares parameters with the policy, its value is normally detached in the actor loss. Otherwise, automatic differentiation adds a gradient through the baseline itself, which is not part of the policy-gradient identity above.</p> 652 653<p>The most important baseline is the value function</p> 654 655\[V^{\pi}(s) 656= 657\mathbb{E}_{\tau_{t:T}\sim p^\pi(\tau_{t:T}\mid s_t=s)} 658\left[ 659G_t 660\right].\] 661 662<p>It is the expected future return when the agent is in state (s) and subsequently follows policy (\pi).</p> 663 664<p>Define the action-value function</p> 665 666\[Q^{\pi}(s,a) 667= 668\mathbb{E}_{\tau_{t:T}\sim p^\pi(\tau_{t:T}\mid s_t=s,a_t=a)} 669\left[ 670G_t 671\right]\] 672 673<p>and the advantage</p> 674 675\[\boxed{ 676\color{#16a34a}{A^{\pi}(s,a)} 677= 678Q^{\pi}(s,a)-V^{\pi}(s). 679}\] 680 681<p>The advantage asks a relative question: was this action better or worse than what the policy usually does in this state?</p> 682 683<ul> 684 <li> 685 <p>(A>0): increase the action probability.</p> 686 </li> 687 <li> 688 <p>(A<0): decrease the action probability.</p> 689 </li> 690 <li> 691 <p>(A=0): this sample supplies no first-order preference.</p> 692 </li> 693</ul> 694 695<p>The policy-gradient theorem can therefore be written in its familiar form <sp
695an class="reference">[Sut99]</span>:</p> 696 697\[\nabla_{\theta}J 698= 699\underset{ 700\substack{ 701s_t\sim d^\pi_t\\ 702a_t\sim\pi_\theta(\cdot\mid s_t) 703} 704}{ 705\mathbb{E} 706} 707\left[ 708 709A^{\pi}(s_t,a_t) 710 711\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t) 712 713\right].\] 714 715<h2 id="the-critic-and-generalized-advantage-estimation">The Critic and Generalized Advantage Estimation</h2> 716 717<p>The policy gradient tells us that actions with higher-than-expected returns should become more likely, while actions with lower-than-expected returns should become less likely.</p> 718 719<p>To do this, we need an estimate of the <strong>advantage</strong> of the action that was actually taken.</p> 720 721<p>The true value function is</p> 722 723\[V^\pi(s) 724= 725\mathbb{E}[G_t\mid s_t=s].\] 726 727<p>It tells us the expected return when starting from state (s) and following the current policy.</p> 728 729<p>In practice, we do not know the true value function. PPO therefore learns a parameterized approximation, called the <strong>critic</strong>:</p> 730 731\[V_\psi(s) 732\approx 733V^\pi(s).\] 734 735<p>The critic is usually a neural network with parameters (\psi). Its job is not to choose actions, but to predict how much return we can expect from a state.</p> 736 737<h3 id="monte-carlo-advantage-estimate">Monte Carlo advantage estimate</h3> 738 739<p>Suppose we have collected a trajectory and observed the return from time (t) onward:</p> 740 741\[G_t 742= 743r_t 744+ 745\gamma r_{t+1} 746+ 747\gamma^2 r_{t+2} 748+\cdots.\] 749 750<p>A simple estimate of the advantage is then</p> 751 752\[\hat A_t 753= 754G_t 755- 756V_\psi(s_t).\] 757 758<p>The intuition is straightforward:</p> 759 760<ul> 761 <li>if the observed return is higher than the critic expected, the action performed better than expected;</li> 762 <li>if the observed return is lower than expected, the action performed worse than expected.</li> 763</ul> 764 765<p>However, the Monte Carlo return uses the entire future trajectory. This can make the estimate noisy, especially for long episodes.</p> 766 767<h3 id="one-step-temporal-difference-error">One-step temporal-difference error</h3> 768 769<p>Instead of waiting until the end of the episode, we can use the critic to estimate the future.</p> 770 771<p>Define the <strong>temporal-difference residual</strong> as</p> 772 773\[\delta_t 774= 775r_t 776+ 777\gamma V_\psi(s_{t+1}) 778- 779V_\psi(s_t).\] 780 781<p>This compares two quantities.</p> 782 783<p>The critic predicted</p> 784 785\[V_\psi(s_t).\] 786 787<p>After taking the action, we observe the immediate reward and use the criticâs prediction at the next state:</p> 788 789\[r_t+\gamma V_\psi(s_{t+1}).\] 790 791<p>The difference is therefore</p> 792 793\[\delta_t 794= 795\text{new estimate} 796- 797\text{old estimate}.\] 798 799<p>It can be interpreted as a <strong>prediction error</strong>: the outcome was better or worse than the critic expected.</p> 800 801<p>If the critic were perfect, then</p> 802 803\[V_\psi(s) 804= 805V^\pi(s).\] 806 807<p>In that case, conditioning on the current state and action gives</p> 808 809\[\mathbb{E} 810\left[ 811\delta_t 812\mid 813s_t=s,\, 814a_t=a 815\right] 816= 817A^\pi(s,a).\] 818 819<p>To see why, recall that the action-value function is</p> 820 821\[Q^\pi(s,a) 822= 823\mathbb{E} 824\left[ 825r_t+\gamma V^\pi(s_{t+1}) 826\mid 827s_t=s,\, 828a_t=a 829\right].\] 830 831<p>Therefore,</p> 832 833\[\begin{aligned} 834\mathbb{E} 835\left[ 836\delta_t 837\mid 838s_t=s,\, 839a_t=a 840\right] 841&= 842Q^\pi(s,a)-V^\pi(s) 843\\ 844&= 845A^\pi(s,a). 846\end{aligned}\] 847 848<p>So a one-step TD residual already provides an advantage estimate when the critic is exact.</p> 849 850<p>The problem is that in practice the critic is not exact. We can therefore look further into the future and combine several TD residuals.</p> 851 852<h3 id="multi-step-advantage-estimates">Multi-step advantage estimates</h3> 853 854<p>Consider a sum of (k) consecutive TD residuals:</p> 855 856\[\hat A_t^{(k)} 857= 858\sum_{\ell=0}^{k-1} 859\gamma^\ell 860\delta_{t+\ell}.\] 861 862<p>Substituting the definition of the TD residual gives</p> 863 864\[\hat A_t^{(k)} 865= 866\sum_{\ell=0}^{k-1} 867\gamma^\ell 868\left( 869r_{t+\ell} 870+ 871\gamma V_\psi(s_{t+\ell+1}) 872- 873V_\psi(s_{t+\ell}) 874\right).\] 875 876<p>The value-function terms in the middle cancel each other. What remains is</p> 877 878\[\hat A_t^{(k)} 879= 880- 881V_\psi(s_t) 882+ 883\sum_{\ell=0}^{k-1} 884\gamma^\ell r_{t+\ell} 885+ 886\gamma^k V_\psi(s_{t+k}).\] 887 888<p>
888This is a (k)-step return minus the value estimate at the starting state.</p> 889 890<p>For (k=1), we recover the one-step TD residual:</p> 891 892\[\hat A_t^{(1)} 893= 894\delta_t.\] 895 896<p>At the other extreme, suppose the trajectory terminates after the final reward and the terminal value is zero:</p> 897 898\[V_\psi(s_T)=0.\] 899 900<p>Taking the full remaining trajectory gives</p> 901 902\[\hat A_t^{(T-t)} 903= 904G_t-V_\psi(s_t).\] 905 906<p>Thus, multi-step estimates provide a spectrum between one-step bootstrapping and a full Monte Carlo return.</p> 907 908<h3 id="generalized-advantage-estimation">Generalized Advantage Estimation</h3> 909 910<p><strong>Generalized Advantage Estimation</strong>, or <strong>GAE</strong>, combines TD residuals over multiple future time steps:</p> 911 912\[\boxed{ 913\hat A_t^{\mathrm{GAE}(\gamma,\lambda)} 914= 915\sum_{\ell=0}^{T-t-1} 916(\gamma\lambda)^\ell 917\delta_{t+\ell} 918}\] 919 920<p>The term</p> 921 922\[(\gamma\lambda)^\ell\] 923 924<p>controls how much influence a TD residual has as it gets further away from the current time step.</p> 925 926<p>Expanding the first few terms makes the idea clearer:</p> 927 928\[\hat A_t 929= 930\delta_t 931+ 932\gamma\lambda\delta_{t+1} 933+ 934(\gamma\lambda)^2\delta_{t+2} 935+ 936\cdots.\] 937 938<p>The parameter (\lambda) therefore controls how far into the future we use TD errors.</p> 939 940<p>When</p> 941 942\[\lambda=0,\] 943 944<p>GAE becomes</p> 945 946\[\hat A_t 947= 948\delta_t.\] 949 950<p>We only use the immediate TD residual. This generally gives lower variance, but it relies more heavily on the critic.</p> 951 952<p>When</p> 953 954\[\lambda=1,\] 955 956<p>we obtain</p> 957 958\[\hat A_t 959= 960\sum_{\ell=0}^{T-t-1} 961\gamma^\ell\delta_{t+\ell}.\] 962 963<p>For a terminated trajectory, the TD residuals telescope, giving</p> 964 965\[\hat A_t 966= 967G_t-V_\psi(s_t).\] 968 969<p>Thus, the two extremes are approximately:</p> 970 971\[\lambda=0 972\quad\Longrightarrow\quad 973\text{one-step TD}\] 974 975<p>and</p> 976 977\[\lambda=1 978\quad\Longrightarrow\quad 979\text{Monte Carlo}.\] 980 981<p>Intermediate values of (\lambda) provide a compromise between these two extremes.</p> 982 983<p>A smaller value of (\lambda) puts more weight on short-term TD estimates. This reduces variance but can increase bias when the critic is inaccurate.</p> 984 985<p>A larger value of (\lambda) incorporates more of the actual future trajectory. This reduces reliance on the critic, but generally increases variance.</p> 986 987<h3 id="computing-gae-backwards">Computing GAE backwards</h3> 988 989<p>GAE is usually computed efficiently by working backwards through the trajectory.</p> 990 991<p>Let (d_t) indicate whether the transition at time (t) terminates the episode:</p> 992 993\[d_t 994= 995\begin{cases} 9961, & \text{if the transition terminates the episode},\\ 9970, & \text{otherwise}. 998\end{cases}\] 999 1000<p>The recursive form is</p> 1001 1002\[\boxed{ 1003\hat A_t 1004= 1005\delta_t 1006+ 1007\gamma\lambda 1008(1-d_t) 1009\hat A_{t+1} 1010}\] 1011 1012<p>where</p> 1013 1014\[\delta_t 1015= 1016r_t 1017+ 1018\gamma(1-d_t) 1019V_\psi(s_{t+1}) 1020- 1021V_\psi(s_t).\] 1022 1023<p>The termination mask prevents information from one episode from leaking into the next.</p> 1024 1025<p>This backward recursion is the form commonly used in PPO implementations.</p> 1026 1027<h3 id="actor-and-critic-together">Actor and critic together</h3> 1028 1029<p>The actor and critic therefore have different jobs.</p> 1030 1031<p>The <strong>actor</strong> represents the policy:</p> 1032 1033\[\pi_\theta(a_t\mid s_t).\] 1034 1035<p>It determines which actions are taken and is updated using the estimated advantages.</p> 1036 1037<p>The <strong>critic</strong> represents the value estimate:</p> 1038 1039\[V_\psi(s_t).\] 1040 1041<p>It predicts the expected return and provides the baseline needed to construct lower-variance advantage estimates.</p> 1042 1043<p>During PPO training, the two networks are updated together but optimize different objectives: the actor improves the policy, while the critic improves its prediction of future returns.</p> 1044 1045<p>The overall picture is therefore:</p> 1046 1047\[\boxed{ 1048\text{trajectory} 1049\;\longrightarrow\; 1050\text{TD residuals} 1051\;\longrightarrow\; 1052\text{GAE advantages} 1053\;\longrightarrow\; 1054\text{policy update} 1055}\] 1056 1057<p>while the critic is simultaneously trained to become a better estimate of the value function.</p> 1058 1059<h2 id="probability-ratios">Probability Ratios</h2> 1060 1061<p>Suppose trajectories were sampled from an old policy \(\pi_{\mathrm{old}}\), but we are now optimizing a new policy \(\pi_{\theta}\). At a fixed state,</p> 1062 1063\[\mathbb{E}_{a\sim\pi_{\theta}}[A_{\mathrm{old}}(s,a)] 1064 1065= 1066 1067\sum_a\pi_{\theta}(a\mid s)A_{\mathrm{old}}(s,a).\] 1068 1069<p>Insert the old policy:</p> 1070 1071\[\begin{aligned} 1072 1073\sum_a\pi_{\theta}(a\mid s)A_{\mathrm{old}}(s,a) 1074 1075&= 1076 1077\sum_a 1078 1079\pi_{\mathrm{old}}(a\mid s) 1080 1081\frac{\pi_{\theta}(a\mid s)} 1082 1083{\pi_{\mathrm{old}}(a\mid s)} 1084 1085A_{\mathrm{old}}(s,a). 1086 1087\end{aligned}\] 1088 1089<p>Define the importance ratio</p> 1090 1091\[\boxed{ 1092 1093\color{#2563eb}{\rho_t(\theta)} 1094 1095= 1096 1097\frac{\pi_{\theta}(a_t\mid s_t)} 1098 1099{\pi_{\mathrm{old}}(a_t\mid s_t)}. 1100 1101}\] 1102 1103<p>This produces the surrogate objective</p> 1104 1105\[L^{\mathrm{CPI}}(\theta) 1106 1107= 1108 1109\mathbb{E}_{t\sim\pi_{\mathrm{old}}} 1110 1111[\color{#2563eb}{\rho_t(\theta)}\hat A_t].\] 1112 1113<p>At the start of an update, when \(\theta=\theta_{\mathrm{old}}\), we have \(\rho_t=1\). Moreover,</p> 1114 1115\[\nabla_{\theta}\rho_t 1116 1117= 1118 1119\rho_t\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t),\] 1120 1121<p>so the gradient of the surrogate at the old policy is the ordinary advantage-weighted policy gradient.</p> 1122 1123<p>The ratio corrects the action distribution at states sampled by the old policy. It does not fully correct the state-visitation distribution, which also changes when the policy changes. This is why the surrogate is most reliable for a local policy update. TRPO formalizes this idea with a KL-divergence trust region <span class="reference">[Sch15a]</span>. PPO replaces the constrained second-order update with a simpler clipped first-order objective <span class="reference">[Sch17]</span>.</p> 1124 1125<h2 id="clipped-objective">Clipped Objective</h2> 1126 1127<p>PPO maximizes</p> 1128 1129\[\boxed{ 1130 1131L^{\mathrm{CLIP}}(\theta) 1132 1133= 1134 1135\mathbb{E}_t 1136 1137\left[ 1138 1139\min 1140 1141\left( 1142 1143\rho_t\hat A_t, 1144 1145\operatorname{clip}(\rho_t,1-\epsilon,1+\epsilon)\hat A_t 1146 1147\right) 1148 1149\right]. 1150 1151}\] 1152 1153<p>
1153Optimizers usually minimize a loss, so implementations use</p> 1154 1155\[\mathcal{L}_{\mathrm{policy}} 1156 1157=-L^{\mathrm{CLIP}}.\] 1158 1159<p>The minimum makes the objective pessimistic: once a sampled action has moved far enough in the favorable direction, that sample stops rewarding an even larger move.</p> 1160 1161<h3 id="positive-advantage">Positive advantage</h3> 1162 1163<p>When \(\hat A_t>0\), increasing the action probability is favorable. The objective becomes</p> 1164 1165\[\ell_t 1166 1167= 1168 1169\hat A_t\min(\rho_t,1+\epsilon).\] 1170 1171<p>For \(\rho_t>1+\epsilon\), the sample is saturated and supplies no further policy gradient.</p> 1172 1173<h3 id="negative-advantage">Negative advantage</h3> 1174 1175<p>When \(\hat A_t<0\), decreasing the action probability is favorable. Because multiplication by a negative value reverses the ordering,</p> 1176 1177\[\ell_t 1178 1179= 1180 1181\hat A_t\max(\rho_t,1-\epsilon).\] 1182 1183<p>For \(\rho_t<1-\epsilon\), the sample is saturated and supplies no further policy gradient.</p> 1184 1185<p>Clipping is asymmetric in an important way: it suppresses an excessively large favorable move, not an unfavorable move. If a positive-advantage action becomes less likely, PPO leaves the corrective gradient active.</p> 1186 1187<h2 id="gradient-of-the-ppo-loss">Gradient of the PPO Loss</h2> 1188 1189<p>Ignoring the exact nondifferentiable clipping boundaries, define</p> 1190 1191\[m_t 1192 1193= 1194 1195\mathbf{1}[\hat A_t>0,\rho_t<1+\epsilon] 1196 1197+ 1198 1199\mathbf{1}[\hat A_t<0,\rho_t>1-\epsilon].\] 1200 1201<p>Then</p> 1202 1203\[\boxed{ 1204 1205\nabla_{\theta}L^{\mathrm{CLIP}} 1206 1207= 1208 1209\mathbb{E}_t 1210 1211\left[ 1212 1213 m_t\rho_t\hat A_t 1214 1215\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t) 1216 1217\right]. 1218 1219}\] 1220 1221<p>The minimized policy loss has the opposite sign:</p> 1222 1223\[\boxed{ 1224 1225\nabla_{\theta}\mathcal{L}_{\mathrm{policy}} 1226 1227=- 1228 1229\mathbb{E}_t 1230 1231\left[ 1232 1233 m_t\rho_t\hat A_t 1234 1235\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t) 1236 1237\right]. 1238 1239}\] 1240 1241<p>This is still advantage-weighted log-likelihood. PPO only changes the coefficient by adding the old-to-new ratio and the clipping mask.</p> 1242 1243<h3 id="gradient-with-respect-to-logits">Gradient with respect to logits</h3> 1244 1245<p>Suppose the policy produces logits \(z_j\) and probabilities</p> 1246 1247\[\pi_j=\frac{e^{z_j}}{\sum_k e^{z_k}}.\] 1248 1249<p>For the sampled action \(a\),</p> 1250 1251\[\frac{\partial\log\pi(a\mid s)}{\partial z_j} 1252 1253= 1254 1255\mathbf{1}[j=a]-\pi_j.\] 1256 1257<p>In an active, unclipped region,</p> 1258 1259\[\frac{\partial\mathcal{L}_{\mathrm{policy},t}}{\partial z_j} 1260 1261= 1262 1263-\rho_t\hat A_t 1264 1265\left(\mathbf{1}[j=a_t]-\pi_j\right).\] 1266 1267<p>For positive advantage, gradient descent raises the selected actionâs logit relative to the alternatives. For negative advantage, it lowers it.</p> 1268 1269<h2 id="the-full-ppo-loss">The Full PPO Loss</h2> 1270 1271<p>A common minimized PPO objective is</p> 1272 1273\[\boxed{ 1274 1275\color{#7c3aed}{\mathcal{L}_{\mathrm{PPO}}} 1276 1277= 1278 1279\mathcal{L}_{\mathrm{policy}} 1280 1281+c_V\mathcal{L}_{\mathrm{value}} 1282 1283-c_H\mathcal{H}. 1284 1285}\] 1286 1287<p>The value loss is often</p> 1288 1289\[\mathcal{L}_{\mathrm{value}} 1290 1291= 1292 1293\frac{1}{2} 1294 1295\mathbb{E}_t 1296 1297\left[ 1298 1299\left(V_{\psi}(s_t)-\hat R_t\right)^2 1300 1301\right],\] 1302 1303<p>where a common target is</p> 1304 1305\[\hat R_t=\hat A_t+V_{\mathrm{old}}(s_t).\] 1306 1307<p>The entropy term</p> 1308 1309\[\mathcal{H} 1310 1311= 1312 1313\mathbb{E}_s 1314 1315\left[ 1316 1317-\sum_a\pi_{\theta}(a\mid s) 1318 1319\log\pi_{\theta}(a\mid s) 1320 1321\right]\] 1322 1323<p>encourages exploration when it is subtracted from the minimized loss.</p> 1324 1325<p>During the actor update, the old log-probabilities, returns, and advantages are treated as constants. The critic is trained through its separate value loss.</p> 1326 1327<h2 id="ppo-for-language-models">PPO for Language Models</h2> 1328 1329<p>For an autoregressive language model,</p> 1330 1331\[s_t=(q,o_{<t}),\qquad a_t=o_t.\] 1332 1333<p>The token-level PPO ratio is</p> 1334 1335\[\rho_t 1336 1337= 1338 1339\frac{ 1340 1341\pi_{\theta}(o_t\mid q,o_{<t}) 1342 1343}{ 1344 1345\pi_{\mathrm{old}}(o_t\mid q,o_{<t}) 1346 1347}.\] 1348 1349<p>It is evaluated stably in log-space:</p> 1350 1351\[\rho_t 1352 1353= 1354 1355\exp 1356 1357\left( 1358 1359\log\pi_{\theta}(o_t\mid q,o_{<t}) 1360 1361- 1362 1363\log\pi_{\mathrm{old}}(o_t\mid q,o_{<t}) 1364 1365\right).\] 1366 1367<p>A sequence-level ratio would multiply all token ratios. For long responses this product can become extremely large or small. Token-level PPO instead views every prefix as a state and every next token as an action.</p> 1368 1369<p>Three models must be kept conceptually separate:</p> 1370 1371<p>1. The current policy \(\pi_{\theta}\) is being optimized.</p> 1372 1373<p>2. The old policy \(\pi_{\mathrm{old}}\) generated the current rollouts and appears in the PPO ratio.</p> 1374 1375<p>3. The reference policy \(\pi_{\mathrm{ref}}\) regularizes behavior toward a stable language model, often the supervised fine-tuned model.</p> 1376 1377<p>PPO also normally uses a fourth component, the critic \(V_{\psi}\). The old policy and reference policy may initially contain identical weights, but they have different mathematical roles. The old policy is part of importance weighting; the reference policy is part of behavioral regularization.</p> 1378 1379<p>For language-model RL, the task reward often arrives only after the complete answer. The critic attempts to predict the expected final outcome from each partial response. This can provide token-specific credit assignment, but it requires training and storing an additional value model. GRPO removes that component.</p> 1380 1381<h2 id="group-relative-policy-optimization">Group Relative Policy Optimization</h2> 1382 1383<p>GRPO was introduced as a critic-free variant of PPO in DeepSeekMath <span class="reference">[Sha24]</span>. For each prompt \(q\), sample a group of \(G\) responses from the old policy:</p> 1384 1385\[o_1,\ldots,o_G 1386 1387\sim 1388 1389\pi_{\mathrm{old}}(\cdot\mid q).\] 1390 1391<p>Score them with a reward function:</p> 1392 1393\[R_1,\ldots,R_G.\] 1394 1395<p>Compute prompt-local statistics</p> 1396 1397\[\mu_q 1398 1399= 1400 1401\frac{1}{G}\sum_{i=1}^G R_i,\] 1402 1403\[\sigma_q 1404 1405= 1406 1407\sqrt{ 1408 1409\frac{1}{G}\sum_{i=1}^G(R_i-\mu_q)^2 1410 1411+\varepsilon_{\mathrm{std}} 1412 1413},\] 1414 1415<p>and define the group-relative advantage</p> 1416 1417\[\boxed{ 1418 1419\hat A_i 1420 1421= 1422 1423\frac{R_i-\mu_q}{\sigma_q}. 1424 1425}\] 1426 1427<p>Under outcome supervision, the original GRPO formulation assigns the same response-level advantage to every token in the response:</p> 1428 1429\[\hat A_{i,t}=\hat A_i.\] 1430 1431<p>The clipped policy term is then the same as in PPO:</p> 1432 1433\[\min 1434 1435\left( 1436 1437\rho_{i,t}\hat A_i, 1438 1439\operatorname{clip}(\rho_{i,t},1-\epsilon,1+\epsilon)\hat A_i 1440 1441\right),\] 1442 1443<p>where</p> 1444 1445\[\rho_{i,t} 1446 1447= 1448 1449\frac{ 1450 1451\pi_{\theta}(o_{i,t}\mid q,o_{i,\lt t}) 1452 1453}{ 1454 1455\pi_{\mathrm{old}}(o_{i,t}\mid q,o_{i,\lt t}) 1456 1457}.\] 1458 1459<p>A common response-normalized form of the objective is</p> 1460 1461\[\begin{aligned} 1462 1463J_{\mathrm{GRPO}}(\theta) 1464 1465= 1466 1467\mathbb{E} 1468 1469\Bigg[ 1470 1471\frac{1}{G} 1472 1473\sum_{i=1}^G 1474 1475\frac{1}{T_i} 1476 1477\sum_{t=1}^{T_i} 1478 1479\Bigg(& 1480 1481\min 1482 1483\left( 1484 1485\rho_{i,t}\hat A_i, 1486 1487\operatorname{clip}(\rho_{i,t},1-\epsilon,1+\epsilon)\hat A_i 1488 1489\right) 1490 1491\\\\ 1492 1493&-\beta\hat D_{\mathrm{KL},i,t} 1494 1495\Bigg) 1496 1497\Bigg]. 1498 1499\end{aligned}\] 1500 1501<p>The factor \(1/T_i\) gives each response equal total weight rather than giving longer responses more weight merely because they contain more tokens.</p> 1502 1503<h2 id="grpos-kl-term">GRPOâs KL Term</h2> 1504 1505<p>The original formulation uses the per-token estimator</p> 1506 1507\[\boxed{ 1508 1509\hat D_{\mathrm{KL}} 1510 1511= 1512 1513\frac{\pi_{\mathrm{ref}}(a\mid s)} 1514 1515{\pi_{\theta}(a\mid s)} 1516 1517- 1518 1519\log 1520 1521\frac{\pi_{\mathrm{ref}}(a\mid s)} 1522 1523{\pi_{\theta}(a\mid s)} 1524 1525-1. 1526 1527}\] 1528 1529<p>Let</p> 1530 1531\[x=\frac{\pi_{\mathrm{ref}}(a\mid s)} 1532 1533{\pi_{\theta}(a\mid s)}.\] 1534 1535<p>Then</p> 1536 1537\[\hat D_{\mathrm{KL}}=x-\log x-1\geq 0,\] 1538 1539<p>
1539because \(\log x\leq x-1\) for \(x>0\).</p> 1540 1541<p>If the action is sampled from the current policy, the expected value is the reverse KL divergence:</p> 1542 1543\[\begin{aligned} 1544 1545\mathbb{E}_{a\sim\pi_{\theta}}[\hat D_{\mathrm{KL}}] 1546 1547&= 1548 1549\sum_a\pi_{\theta}(a) 1550 1551\left[ 1552 1553\frac{\pi_{\mathrm{ref}}(a)}{\pi_{\theta}(a)} 1554 1555- 1556 1557\log\frac{\pi_{\mathrm{ref}}(a)}{\pi_{\theta}(a)} 1558 1559-1 1560 1561\right] 1562 1563\\\\ 1564 1565&= 1566 1567D_{\mathrm{KL}}\left(\pi_{\theta}\,\|\,\pi_{\mathrm{ref}}\right) 1568 1569\end{aligned}\] 1570 1571<p>For a fixed sampled token, define</p> 1572 1573\[u 1574 1575= 1576 1577\log\pi_{\theta}(a\mid s) 1578 1579- 1580 1581\log\pi_{\mathrm{ref}}(a\mid s).\] 1582 1583<p>Then \(x=e^{-u}\) and</p> 1584 1585\[\frac{\partial\hat D_{\mathrm{KL}}}{\partial u} 1586 1587=1-e^{-u}.\] 1588 1589<p>Therefore,</p> 1590 1591\[\boxed{ 1592 1593\nabla_{\theta}\hat D_{\mathrm{KL}} 1594 1595= 1596 1597\left( 1598 15991- 1600 1601\frac{\pi_{\mathrm{ref}}(a\mid s)} 1602 1603{\pi_{\theta}(a\mid s)} 1604 1605\right) 1606 1607\nabla_{\theta}\log\pi_{\theta}(a\mid s). 1608 1609}\] 1610 1611<p>The policy-gradient portion of GRPO is otherwise identical to PPO. Its active gradient is</p> 1612 1613\[\boxed{ 1614 1615\nabla_{\theta}J_{\mathrm{GRPO,policy}} 1616 1617= 1618 1619\mathbb{E} 1620 1621\left[ 1622 1623\frac{1}{G}\sum_i\frac{1}{T_i}\sum_t 1624 1625m_{i,t}\rho_{i,t}\hat A_i 1626 1627\nabla_{\theta} 1628 1629\log\pi_{\theta}(o_{i,t}\mid q,o_{i,\lt t}) 1630 1631\right]. 1632 1633}\] 1634 1635<p>The algorithmic difference is where \(\hat A\) comes from:</p> 1636 1637\[\text{PPO: reward-to-go minus a learned value baseline,}\] 1638 1639\[\text{GRPO: response reward minus a same-prompt group baseline.}\] 1640 1641<h2 id="what-group-normalization-changes">What Group Normalization Changes</h2> 1642 1643<p>Group normalization has several consequences that are easy to miss.</p> 1644 1645<h3 id="the-advantages-are-centered">The advantages are centered</h3> 1646 1647<p>Ignoring the numerical epsilon,</p> 1648 1649\[\sum_{i=1}^G\hat A_i=0.\] 1650 1651<p>GRPO therefore performs a relative update within each prompt. Some responses are pushed up and others are pushed down.</p> 1652 1653<h3 id="positive-affine-reward-transformations-disappear">Positive affine reward transformations disappear</h3> 1654 1655<p>For \(R_i'=aR_i+b\) with \(a>0\),</p> 1656 1657\[\frac{R_i'-\mu'}{\sigma'} 1658 1659= 1660 1661\frac{R_i-\mu}{\sigma}.\] 1662 1663<p>This makes GRPO insensitive to prompt-specific reward offsets and positive scales. It also means absolute reward magnitude is discarded after normalization.</p> 1664 1665<h3 id="equal-rewards-produce-no-reward-gradient">Equal rewards produce no reward gradient</h3> 1666 1667<p>If all responses receive the same reward, then every centered advantage is zero. The prompt supplies no reward-driven policy update. This can happen when all sampled answers are wrong, all are correct, or the reward function is too coarse.</p> 1668 1669<h3 id="with-two-samples-grpo-is-close-to-ranking">With two samples, GRPO is close to ranking</h3> 1670 1671<p>With \(G=2\), population standard deviation, and unequal rewards, the normalized advantages are \(+1\) and \(-1\). Only the ordering remains; the size of the reward gap disappears. A different standard-deviation convention changes the magnitude but not the direction.</p> 1672 1673<h3 id="the-ordinary-group-mean-contains-the-current-sample">The ordinary group mean contains the current sample</h3> 1674 1675<p>A textbook action-independent baseline should not depend on the sampled action whose score it multiplies. The group mean</p> 1676 1677\[\bar R=\frac{1}{G}\sum_jR_j\] 1678 1679<p>contains \(R_i\) itself. For the unnormalized centered estimator</p> 1680 1681\[\hat g 1682 1683= 1684 1685\frac{1}{G} 1686 1687\sum_{i=1}^G 1688 1689(R_i-\bar R) 1690 1691\nabla\log\pi(o_i),\] 1692 1693<p>independent sampling gives</p> 1694 1695\[\mathbb{E}[\hat g] 1696 1697= 1698 1699\frac{G-1}{G} 1700 1701\mathbb{E} 1702 1703\left[ 1704 1705R\nabla\log\pi(o) 1706 1707\right].\] 1708 1709<p>Thus self-inclusion preserves the expected direction but scales the unnormalized estimator. A leave-one-out baseline</p> 1710 1711\[b_{-i} 1712 1713= 1714 1715\frac{1}{G-1}\sum_{j\neq i}R_j\] 1716 1717<p>is independent of response \(i\) conditional on the prompt and removes this particular factor. Dividing by the random group standard deviation introduces additional stochastic rescaling, so normalized GRPO is best understood as a practical relative estimator rather than a literal unbiased copy of REINFORCE.</p> 1718 1719<h2 id="a-small-numerical-grpo-example">A Small Numerical GRPO Example</h2> 1720 1721<p>Suppose four responses to the same prompt receive</p> 1722 1723\[R=[1,0,0,0].\] 1724 1725<p>The group mean is</p> 1726 1727\[\mu=0.25.\] 1728 1729<p>Using population standard deviation,</p> 1730 1731\[\sigma 1732 1733= 1734 1735\sqrt{ 1736 1737\frac{(0.75)^2+3(-0.25)^2}{4} 1738 1739} 1740 1741\approx0.433.\] 1742 1743<p>The normalized advantages are</p> 1744 1745\[\hat A 1746 1747\approx 1748 1749[1.732,-0.577,-0.577,-0.577].\] 1750 1751<p>Every token in the successful response receives positive advantage, while tokens in the three unsuccessful responses receive negative advantage.</p> 1752 1753<p>Now let \(\epsilon=0.2\). For a token in the successful response with \(\rho=1.10\),</p> 1754 1755\[\rho\hat A 1756 1757=1.10\times1.732 1758 1759\approx1.905,\] 1760 1761<p>
1761and the gradient is active. If \(\rho=1.25\), PPO clipping limits the objective contribution to</p> 1762 1763\[1.2\times1.732 1764 1765\approx2.078,\] 1766 1767<p>and that sample supplies no further favorable policy gradient beyond the clipping boundary.</p> 1768 1769<h2 id="ppo-and-grpo-side-by-side">PPO and GRPO Side by Side</h2> 1770 1771<table> 1772 <thead> 1773 <tr> 1774 <th>Property</th> 1775 <th>PPO</th> 1776 <th>GRPO</th> 1777 </tr> 1778 </thead> 1779 <tbody> 1780 <tr> 1781 <td>Policy update</td> 1782 <td>Clipped probability-ratio objective</td> 1783 <td>Same clipped objective</td> 1784 </tr> 1785 <tr> 1786 <td>Baseline</td> 1787 <td>Learned value function</td> 1788 <td>Same-prompt group statistics</td> 1789 </tr> 1790 <tr> 1791 <td>Critic</td> 1792 <td>Normally required</td> 1793 <td>Not required</td> 1794 </tr> 1795 <tr> 1796 <td>Advantage granularity</td> 1797 <td>Usually token- or time-specific through GAE</td> 1798 <td>Often one outcome advantage per response</td> 1799 </tr> 1800 <tr> 1801 <td>Value loss</td> 1802 <td>Yes</td> 1803 <td>No</td> 1804 </tr> 1805 <tr> 1806 <td>Rollouts</td> 1807 <td>Ordinary trajectories</td> 1808 <td>Multiple responses for each prompt</td> 1809 </tr> 1810 <tr> 1811 <td>Main benefit</td> 1812 <td>Fine-grained learned credit assignment</td> 1813 <td>Lower memory and implementation complexity</td> 1814 </tr> 1815 <tr> 1816 <td>Main weakness</td> 1817 <td>Critic cost and value-estimation error</td> 1818 <td>Coarse credit assignment and zero-signal groups</td> 1819 </tr> 1820 </tbody> 1821</table> 1822 1823<p>A compact summary is</p> 1824 1825\[\boxed{ 1826 1827\text{GRPO is a PPO-style update with a prompt-local Monte Carlo baseline.} 1828 1829}\] 1830 1831<p>It removes the critic. It does not remove the reward function, the rollout policy, PPO clipping, or necessarily the reference policy.</p> 1832 1833<h2 id="minimal-pytorch-style-implementation">Minimal PyTorch-Style Implementation</h2> 1834 1835<p>The following code contains the mathematical core. Production implementations additionally need distributed rollout collection, padding masks, mixed precision, optimizer state management, and careful metric logging.</p> 1836 1837<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code> 1838<span class="kn">from</span> <span class="nn">__future__</span> <span class="kn">import</span> <span class="n">annotations</span> 1839 1840<span class="kn">import</span> <span class="nn">torch</span> 1841 1842<span class="kn">from</span> <span class="nn">torch</span> <span class="kn">import</span> <span class="n">Tensor</span> 1843 1844 1845 1846<span class="k">def</span> <span class="nf">ppo_policy_loss</span><span class="p">(</span> 1847 1848 <span class="n">log_probs</span><span class="p">:</span> <span class="n">Tensor</span><span class="p">,</span> 1849 1850 <span class="n">old_log_probs</span><span class="p">:</span> <span class="n">Tensor</span><span class="p">,</span> 1851 1852 <span class="n">advantages</span><span class="p">:</span> <span class="n">Tensor</span><span class="p">,</span> 1853 1854 <span class="n">mask</span><span class="p">:</span> <span class="n">Tensor</span><span class="p">,</span> 1855 1856 \<span class="o">*</span><span class="p">,</span> 1857 1858 <span class="n">clip_epsilon</span><span class="p">:</span> <span class="nb">float</span> <span class="o">=</span> <span class="mf">0.2</span><span class="p">,</span> 1859 1860<span class="p">)</span> <span class="o">-></span> <span class="n">Tensor</span><span class="p">:</span> 1861 1862 <span class="s">"""Clipped PPO policy loss over valid tokens."""</span> 1863 1864 <span class="n">old_log_probs</span> <span class="o">=</span> <span class="n">old_log_probs</span><span class="p">.</span><span class="n">detach</span><span class="p">()</span> 1865 1866 <span class="n">advantages</span> <span class="o">=</span> <span class="n">advantages</span><span class="p">.</span><span class="n">detach</span><span class="p">()</span> 1867 1868 <span class="n">mask</span> <span class="o">=</span> <span class="n">mask</span><span class="p">.</span><span class="n">to</span><span class="p">(</span><span class="n">log_probs</span><span class="p">.</span><span class="n">dtype</span><span class="p">)</span> 1869 1870 <span class="n">ratio</span> <span class="o">=</span> <span class="n">torch</span><span class="p">.</span><span class="n">exp</span><span class="p">(</span><span class="n">log_probs</span> <span class="o">-</span> <span class="n">old_log_probs</span><span class="p">)</span> 1871 1872 <span class="n">unclipped</span> <span class="o">=</span> <span class="n">ratio</span> \<span class="o">*</span> <span class="n">advantages</span> 1873 1874 <span class="n">clipped</span> <span class="o">=</span> <span class="n">ratio</span><span class="p">.</span><span class="n">clamp</span><span class="p">(</span> 1875
1876 <span class="mf">1.0</span> <span class="o">-</span> <span class="n">clip_epsilon</span><span class="p">,</span> 1877 1878 <span class="mf">1.0</span> <span class="o">+</span> <span class="n">clip_epsilon</span><span class="p">,</span> 1879 1880 <span class="p">)</span> \<span class="o">*</span> <span class="n">advantages</span> 1881 1882 <span class="n">token_objective</span> <span class="o">=</span> <span class="n">torch</span><span class="p">.</span><span class="n">minimum</span><span class="p">(</span><span class="n">unclipped</span><span class="p">,</span> <span class="n">clipped</span><span class="p">)</span> 1883 1884 <span class="n">valid_tokens</span> <span class="o">=</span> <span class="n">mask</span><span class="p">.</span><span class="nb">sum</span><span class="p">().</span><span class="n">clamp_min</span><span class="p">(</span><span class="mf">1.0</span><span class="p">)</span> 1885 1886 <span class="k">return</span> <span class="o">-</span><span class="p">(</span><span class="n">token_objective</span> \<span class="o">*</span> <span class="n">mask</span><span class="p">).</span><span class="nb">sum</span><span class="p">()</span> <span class="o">/</span> <span class="n">valid_tokens</span> 1887 1888 1889 1890<span class="k">def</span> <span class="nf">grpo_loss</span><span class="p">(</span> 1891 1892 <span class="n">log_probs</span><span class="p">:</span> <span class="n">Tensor</span><span class="p">,</span> <span class="c1"># [batch, group, time] 1893</span> 1894 <span class="n">old_log_probs</span><span class="p">:</span> <span class="n">Tensor</span><span class="p">,</span> <span class="c1"># [batch, group, time] 1895</span> 1896 <span class="n">ref_log_probs</span><span class="p">:</span> <span class="n">Tensor</span><span class="p">,</span> <span class="c1"># [batch, group, time] 1897</span> 1898 <span class="n">rewards</span><span class="p">:</span> <span class="n">Tensor</span><span class="p">,</span> <span class="c1"># [batch, group] 1899</span> 1900 <span class="n">mask</span><span class="p">:</span> <span class="n">Tensor</span><span class="p">,</span> <span class="c1"># [batch, group, time] 1901</span> 1902 \<span class="o">*</span><span class="p">,</span> 1903 1904 <span class="n">clip_epsilon</span><span class="p">:</span> <span class="nb">float</span> <span class="o">=</span> <span class="mf">0.2</span><span class="p">,</span> 1905 1906 <span class="n">kl_coefficient</span><span class="p">:</span> <span class="nb">float</span> <span class="o">=</span> <span class="mf">0.01</span><span class="p">,</span> 1907 1908 <span class="n">std_epsilon</span><span class="p">:</span> <span class="nb">float</span> <span class="o">=</span> <span class="mf">1e-6</span><span class="p">,</span> 1909 1910<span class="p">)</span> <span class="o">-></span> <span class="n">Tensor</span><span class="p">:</span> 1911 1912 <span class="s">"""Outcome-supervised, response-normalized GRPO loss."""</span> 1913 1914 <span class="n">old_log_probs</span> <span class="o">=</span> <span class="n">old_log_probs</span><span class="p">.</span><span class="n">detach</span><span class="p">()</span> 1915 1916 <span class="n">ref_log_probs</span> <span class="o">=</span> <span class="n">ref_log_probs</span><span class="p">.</span><span class="n">detach</span><span class="p">()</span> 1917 1918 <span class="n">rewards</span> <span class="o">=</span> <span class="n">rewards</span><span class="p">.</span><span class="n">detach</span><span class="p">()</span> 1919 1920 <span class="n">mask</span> <span class="o">=</span> <span class="n">mask</span><span class="p">.</span><span class="n">to</span><span class="p">(</span><span class="n">log_probs</span><span class="p">.</span><span class="n">dtype</span><span class="p">)</span> 1921 1922 <span class="n">group_mean</span> <span class="o">=</span> <span class="n">rewards</span><span class="p">.</span><span class="n">mean</span><span class="p">(</span><span class="n">dim</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">keepdim</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span> 1923 1924 <span class="n">group_variance</span> <span class="o">=</span> <span class="p">(</span><span class="n">rewards</span> <span class="o">-</span> <span class="n">group_mean</span><span class="p">).</span><span class="n">square</span><span class="p">().</span><span class="n">mean</span><span class="p">(</span> 1925
1926 <span class="n">dim</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> 1927 1928 <span class="n">keepdim</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span> 1929 1930 <span class="p">)</span> 1931 1932 <span class="n">group_std</span> <span class="o">=</span> <span class="n">torch</span><span class="p">.</span><span class="n">sqrt</span><span class="p">(</span><span class="n">group_variance</span> <span class="o">+</span> <span class="n">std_epsilon</span><span class="p">)</span> 1933 1934 <span class="n">response_advantage</span> <span class="o">=</span> <span class="p">(</span> 1935 1936 <span class="p">(</span><span class="n">rewards</span> <span class="o">-</span> <span class="n">group_mean</span><span class="p">)</span> <span class="o">/</span> <span class="n">group_std</span> 1937 1938 <span class="p">).</span><span class="n">detach</span><span class="p">()</span> 1939 1940 <span class="n">token_advantage</span> <span class="o">=</span> <span class="n">response_advantage</span><span class="p">.</span><span class="n">unsqueeze</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">)</span> 1941 1942 <span class="n">ratio</span> <span class="o">=</span> <span class="n">torch</span><span class="p">.</span><span class="n">exp</span><span class="p">(</span><span class="n">log_probs</span> <span class="o">-</span> <span class="n">old_log_probs</span><span class="p">)</span> 1943 1944 <span class="n">unclipped</span> <span class="o">=</span> <span class="n">ratio</span> \<span class="o">*</span> <span class="n">token_advantage</span> 1945 1946 <span class="n">clipped</span> <span class="o">=</span> <span class="n">ratio</span><span class="p">.</span><span class="n">clamp</span><span class="p">(</span> 1947 1948 <span class="mf">1.0</span> <span class="o">-</span> <span class="n">clip_epsilon</span><span class="p">,</span> 1949 1950 <span class="mf">1.0</span> <span class="o">+</span> <span class="n">clip_epsilon</span><span class="p">,</span> 1951 1952 <span class="p">)</span> \<span class="o">*</span> <span class="n">token_advantage</span> 1953 1954 <span class="n">policy_token_loss</span> <span class="o">=</span> <span class="o">-</span><span class="n">torch</span><span class="p">.</span><span class="n">minimum</span><span class="p">(</span><span class="n">unclipped</span><span class="p">,</span> <span class="n">clipped</span><span class="p">)</span> 1955 1956 <span class="n">log_ref_over_policy</span> <span class="o">=</span> <span class="n">ref_log_probs</span> <span class="o">-</span> <span class="n">log_probs</span> 1957 1958 <span class="n">kl_token_loss</span> <span class="o">=</span> <span class="p">(</span> 1959 1960 <span class="n">torch</span><span class="p">.</span><span class="n">exp</span><span class="p">(</span><span class="n">log_ref_over_policy</span><span class="p">)</span> 1961 1962 <span class="o">-</span> <span class="n">log_ref_over_policy</span> 1963 1964 <span class="o">-</span> <span class="mf">1.0</span> 1965 1966 <span class="p">)</span> 1967 1968 <span class="n">token_loss</span> <span class="o">=</span> <span class="n">policy_token_loss</span> <span class="o">+</span> <span class="n">kl_coefficient</span> \<span class="o">*</span> <span class="n">kl_token_loss</span> 1969 1970 <span class="n">token_count</span> <span class="o">=</span> <span class="n">mask</span><span class="p">.</span><span class="nb">sum</span><span class="p">(</span><span class="n">dim</span><span class="o">=-</span><span class="mi">1</span><span class="p">).</span><span class="n">clamp_min</span><span class="p">(</span><span class="mf">1.0</span><span class="p">)</span> 1971 1972 <span class="n">response_loss</span> <span class="o">=</span> <span class="p">(</span><span class="n">token_loss</span> \<span class="o">*</span> <span class="n">mask</span><span class="p">).</span><span class="nb">sum</span><span class="p">(</span><span class="n">dim</span><span class="o">=-</span><span class="mi">1</span><span class="p">)</span> <span class="o">/</span> <span class="n">token_count</span> 1973 1974 <span class="k">return</span> <span class="n">response_loss</span><span class="p">.</span><span class="n">mean</span><span class="p">()</span> 1975 1976</code></pre></div></div> 1977 1978<p>The essential implementation rules are visible in the code:</p> 1979 1980<ul> 1981 <li> 1982 <p>Compute probability ratios as exponentiated log-probability differences.</p> 1983 </li> 1984 <li> 1985 <p>Detach old log-probabilities and advantages.</p> 1986 </li> 1987 <li> 1988 <p>Normalize GRPO rewards independently within each prompt group.</p> 1989 </li> 1990 <li> 1991 <p>Apply the response mask before averaging.</p> 1992 </li> 1993 <li> 1994 <p>Decide deliberately whether to weight responses equally or tokens equally.</p> 1995 </li> 1996</ul> 1997 1998<h2 id="common-failure-modes">Common Failure Modes</h2> 1999 2000<h3 id="confusing-objective-and-loss-signs">Confusing objective and loss signs</h3> 2001 2002<p>Papers usually write an objective to maximize. Deep-learning libraries usually minimize. The implementation therefore needs a leading minus sign on the clipped policy objective.</p> 2003 2004<h3 id="dividing-log-probabilities">Dividing log-probabilities</h3> 2005 2006<p>The correct ratio is</p> 2007 2008\[\rho=\exp(\log\pi_{\theta}-\log\pi_{\mathrm{old}}),\] 2009 2010<p>not \(\log\pi_{\theta}/\log\pi_{\mathrm{old}}\).</p> 2011 2012<h3 id="backpropagating-through-rollout-quantities">Backpropagating through rollout quantities</h3> 2013 2014<p>Old log-probabilities, sampled rewards, returns, and advantages are fixed data during the actor update. They should not retain a gradient path to trainable rollout models.</p> 2015 2016<h3 id="confusing-the-old-and-reference-policies">Confusing the old and reference policies</h3> 2017 2018<p>The old policy is the denominator of the PPO ratio. The reference policy is the target of KL regularization. Replacing one role with the other changes the algorithm.</p> 2019 2020<h3 id="refreshing-the-old-policy-inside-the-optimization-epoch">Refreshing the old policy inside the optimization epoch</h3> 2021 2022<p>The old policy should remain fixed while the rollout batch is reused for several minibatch updates. Otherwise the denominator changes and the clipping interpretation no longer matches the collected data.</p> 2023 2024<h3 id="treating-clipping-as-a-global-kl-guarantee">Treating clipping as a global KL guarantee</h3> 2025 2026<p>PPO clipping limits the incentive contributed by individual sampled actions. It does not mathematically guarantee that the complete policy remains inside a fixed KL ball. Learning rate, number of epochs, and correlated parameter changes still matter.</p> 2027 2028<h3 id="normalizing-grpo-rewards-across-unrelated-prompts">Normalizing GRPO rewards across unrelated prompts</h3> 2029 2030<p>The defining comparison is within the group generated for one prompt. Batch-wide normalization mixes prompt difficulty and changes the meaning of the advantage.</p> 2031 2032<h3 id="ignoring-zero-variance-groups">Ignoring zero-variance groups</h3> 2033 2034<p>When every response in a group receives the same reward, the normalized advantages vanish. Monitoring the fraction of zero-variance groups is therefore an important GRPO diagnostic.</p> 2035 2036<h2 id="a-unifying-view-of-post-training">A Unifying View of Post-Training</h2> 2037 2038<p>Many post-training methods can be understood as weighted log-likelihood updates:</p> 2039 2040\[\nabla_{\theta}J 2041 2042= 2043 2044\mathbb{E} 2045 2046\left[ 2047 2048 c(q,o,t) 2049 2050\nabla_{\theta} 2051 2052\log\pi_{\theta}(o_t\mid q,o_{<t}) 2053 2054\right].\] 2055 2056<p>The coefficient \(c\) defines the method.</p> 2057 2058<p>For supervised fine-tuning,</p> 2059 2060\[c=1\] 2061 2062<p>on selected demonstration tokens.</p> 2063 2064<p>For REINFORCE,</p> 2065 2066\[c=G_t.\] 2067 2068<p>For advantage actor-critic methods,</p> 2069 2070\[c=\hat A_t.\] 2071 2072<p>For active PPO samples,</p> 2073 2074\[c=m_t\rho_t\hat A_t.\] 2075 2076<p>For outcome-supervised GRPO,</p> 2077 2078\[c=m_{i,t}\rho_{i,t} 2079 2080\frac{R_i-\mu_q}{\sigma_q}.\] 2081 2082<p>This perspective makes the algorithms less mysterious. The neural-network backpropagation is ordinary log-likelihood backpropagation. Most of the conceptual work lies in constructing a useful, low-variance, stable coefficient for every sampled token.</p> 2083 2084<h2 id="conclusion">Conclusion</h2> 2085 2086<p>The path from policy gradients to PPO and GRPO consists of a small number of reusable ideas.</p> 2087 2088<p>First, the log-derivative trick turns the gradient of expected reward into reward-weighted log-probability gradients. Second, causality removes past rewards: for \(k\lt t\),</p> 2089 2090\[\mathbb{E} 2091 2092\left[ 2093 2094 r_k\nabla\log\pi(a_t\mid s_t) 2095 2096\right]=0,\] 2097 2098<p>because the expected score of the newly sampled action is zero after conditioning on the history. This produces reward-to-go.</p> 2099 2100<p>Third, the same score identity permits action-independent baselines, leading to advantages. PPO estimates those advantages with a critic, often through GAE, and stabilizes data reuse with an old-policy ratio and clipping. GRPO keeps the clipped PPO machinery but replaces the learned value baseline with a relative baseline constructed from several responses to the same prompt.</p> 2101 2102<p>At the start of either update, when the current and old policies coincide, the central instruction is simple:</p> 2103 2104\[\boxed{ 2105 2106\text{Increase the probability of positive-advantage tokens and decrease the probability of negative-advantage tokens.} 2107 2108}\] 2109 2110<p>PPO and GRPO differ mainly in how they decide which sampled tokens deserve those positive or negative weights, and in how they prevent repeated o
2110ptimization from moving the policy too aggressively.</p> 2111 2112<h2 id="references">References</h2> 2113 2114<p><span class="reference">[Wil92]</span> R. J. Williams. [Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning](https://link.springer.com/article/10.1007/BF00992696). <em>*Machine Learning*</em>, 1992.</p> 2115 2116<p><span class="reference">[Sut99]</span> R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour. [Policy Gradient Methods for Reinforcement Learning with Function Approximation](https://proceedings.neurips.cc/paper/1999/hash/464d828b85b0bed98e80ade0a5c43b0f-Abstract.html). <em>*NeurIPS*</em>, 1999.</p> 2117 2118<p><span class="reference">[Sch15a]</span> J. Schulman, S. Levine, P. Moritz, M. I. Jordan, and P. Abbeel. [Trust Region Policy Optimization](https://arxiv.org/abs/1502.05477). 2015.</p> 2119 2120<p><span class="reference">[Sch15b]</span> J. Schulman, P. Moritz, S. Levine, M. I. Jordan, and P. Abbeel. [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://arxiv.org/abs/1506.02438). 2015.</p> 2121 2122<p><span class="reference">[Sch17]</span> J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. [Proximal Policy Optimization Algorithms](https://arxiv.org/abs/1707.06347). 2017.</p> 2123 2124<p><span class="reference">[Sha24]</span> Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo. [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://arxiv.org/abs/2402.03300). 2024.</p> 2125 2126 </div><div id="disqus_thread"></div> 2127
2127<script> 2128 var disqus_config = function () { 2129 this.page.url = 'https://www.norange.ai/post-training/2026/08/24/post-training-primer.html'; 2130 this.page.identifier = 'https://www.norange.ai/post-training/2026/08/24/post-training-primer.html'; 2131 }; 2132 2133 (function() { 2134 var d = document, s = d.createElement('script'); 2135 2136 s.src = 'https://nikitadurasov-github-io.disqus.com/embed.js'; 2137 2138 s.setAttribute('data-timestamp', +new Date()); 2139 (d.head || d.body).appendChild(s); 2140 })(); 2141 </script>
2141 2142 <noscript>Please enable JavaScript to view the <a href="https://disqus.com/?ref_noscript" rel="nofollow">comments powered by Disqus.</a></noscript><a class="u-url" href="/post-training/2026/08/24/post-training-primer.html" hidden></a> 2143</article> 2144
2145<script> 2146(function () { 2147 var content = document.querySelector(".post-content"); 2148 var nav = document.querySelector(".post-toc"); 2149 if (!content) return; 2150 2151 var headings = Array.prototype.slice 2152 .call(content.querySelectorAll("h2, h3, h4, h5, h6")) 2153 .filter(function (heading) { 2154 return heading.textContent.trim().length > 0; 2155 }); 2156 2157 if (!headings.length) return; 2158 2159 function slugify(text) { 2160 return text 2161 .toLowerCase() 2162 .replace(/[^\w\s-]/g, "") 2163 .trim() 2164 .replace(/\s+/g, "-"); 2165 } 2166 2167 var usedIds = {}; 2168 2169 // Labels are captured before anchors are appended, so the "##" markers below 2170 // never leak into the table of contents. 2171 var sections = headings.map(function (heading) { 2172 var label = heading.textContent.trim(); 2173 2174 if (!heading.id) { 2175 var base = slugify(label) || "section"; 2176 var id = base; 2177 var suffix = 1; 2178 while (document.getElementById(id) || usedIds[id]) { 2179 id = base + "-" + suffix++; 2180 } 2181 heading.id = id; 2182 } 2183 usedIds[heading.id] = true; 2184 2185 return { 2186 heading: heading, 2187 level: parseInt(heading.tagName.charAt(1), 10), 2188 label: label 2189 }; 2190 }); 2191 2192 function copyText(text) { 2193 if (navigator.clipboard && navigator.clipboard.writeText) { 2194 navigator.clipboard.writeText(text); 2195 return; 2196 } 2197 var field = document.createElement("textarea"); 2198 field.value = text; 2199 field.setAttribute("readonly", ""); 2200 field.style.position = "fixed"; 2201 field.style.opacity = "0"; 2202 document.body.appendChild(field); 2203 field.select(); 2204 try { 2205 document.execCommand("copy"); 2206 } catch (err) {} 2207 document.body.removeChild(field); 2208 } 2209 2210 var copiedTimer; 2211 2212 sections.forEach(function (section) { 2213 var heading = section.heading; 2214 2215 var anchor = document.createElement("a"); 2216 anchor.className = "heading-anchor"; 2217 anchor.href = "#" + heading.id; 2218 anchor.setAttribute("aria-label", "Copy link to this section"); 2219 anchor.textContent = new Array(section.level + 1).join("+"); 2220 2221 heading.firstChild.nodeValue = " " + heading.firstChild.nodeValue; 2222 heading.insertBefore(anchor, heading.firstChild); 2223 heading.classList.add("is-linkable"); 2224 2225 heading.addEventListener("click", function (event) { 2226 if (String(window.getSelection())) return; 2227 event.preventDefault(); 2228 2229 copyText( 2230 location.origin + location.pathname + "#" + heading.id 2231 ); 2232 history.replaceState(null, "", "#" + heading.id); 2233 2234 window.clearTimeout(copiedTimer); 2235 Array.prototype.forEach.call( 2236 content.querySelectorAll(".heading-anchor.is-copied"), 2237 function (node) { 2238 node.classList.remove("is-copied"); 2239 } 2240 ); 2241 anchor.classList.add("is-copied"); 2242 copiedTimer = window.setTimeout(function () { 2243 anchor.classList.remove("is-copied"); 2244 }, 1600); 2245 }); 2246 }); 2247 2248 if (!nav || sections.length < 3) return; 2249 2250 var root = document.createElement("ul"); 2251 2252 var stack = [{ level: 0, list: root }]; 2253 var links = []; 2254 2255 sections.forEach(function (section) { 2256 var heading = section.heading; 2257 var level = section.level; 2258 2259 while (stack.length > 1 && level <= stack[stack.length - 1].level) { 2260 stack.pop(); 2261 } 2262 2263 var item = document.createElement("li"); 2264 2265 var link = document.createElement("a"); 2266 link.href = "#" + heading.id; 2267 link.textContent = section.label; 2268 item.appendChild(link); 2269 2270 stack[stack.length - 1].list.appendChild(item); 2271 2272 var sublist = document.createElement("ul"); 2273 item.appendChild(sublist); 2274 stack.push({ level: level, list: sublist }); 2275 2276 links.push({ heading: heading, link: link, item: item }); 2277 }); 2278 2279 Array.prototype.forEach.call(root.querySelectorAll("ul"), function (list) { 2280 if (!list.children.length) list.parentNode.removeChild(list); 2281 }); 2282 2283 nav.appendChild(root); 2284 nav.classList.add("post-toc--ready"); 2285 2286 var active = null; 2287 2288 function keepVisible(link) { 2289 var navBox = nav.getBoundingClientRect(); 2290 var linkBox = link.getBoundingClientRect(); 2291 if (linkBox.top < navBox.top) { 2292 nav.scrollTop -= navBox.top - linkBox.top + 24; 2293 } else if (linkBox.bottom > navBox.bottom) { 2294 nav.scrollTop += linkBox.bottom - navBox.bottom + 24; 2295 } 2296 } 2297 2298 function highlight() { 2299 var current = links[0]; 2300 for (var i = 0; i < links.length; i++) { 2301 if (links[i].heading.getBoundingClientRect().top > 120) break; 2302 current = links[i]; 2303 } 2304 2305 if (current === active) return; 2306 active = current; 2307 2308 Array.prototype.forEach.call( 2309 nav.querySelectorAll(".is-active, .is-parent"), 2310 function (node) { 2311 node.classList.remove("is-active", "is-parent"); 2312 } 2313 ); 2314 2315 current.link.classList.add("is-active"); 2316 2317 var parent = current.item.parentNode; 2318 while (parent && parent !== root) { 2319 if (parent.tagName === "LI") { 2320 var parentLink = parent.querySelector("a"); 2321 if (parentLink) parentLink.classList.add("is-parent"); 2322 } 2323 parent = parent.parentNode; 2324 } 2325 2326 keepVisible(current.link); 2327 } 2328 2329 var ticking = false;
2330 function onScroll() { 2331 if (ticking) return; 2332 ticking = true; 2333 window.requestAnimationFrame(function () { 2334 ticking = false; 2335 highlight(); 2336 }); 2337 } 2338 2339 window.addEventListener("scroll", onScroll, { passive: true }); 2340 window.addEventListener("resize", onScroll); 2341 highlight(); 2342})(); 2343</script>
2343 2344
2345<script> 2346fetch("/assets/views.json") 2347 .then((res) => res.json()) 2348 .then((data) => { 2349 const path = window.location.pathname.replace(/\/$/, ""); 2350 const basename = path.split("/").pop(); // ex: "uncertainty-estimation-intro.html", sums up all views ending with this basename 2351 2352 let total = 0; 2353 for (const [k, v] of Object.entries(data)) { 2354 if (typeof v === "number" && k.endsWith("/" + basename)) { 2355 total += v; 2356 } 2357 } 2358 2359 document.getElementById("ga_site_pv").textContent = total.toLocaleString(); 2360 }) 2361 .catch((err) => { 2362 console.error("Failed to load total view count", err); 2363 document.getElementById("ga_site_pv").textContent = "N/A"; 2364 }); 2365 </script>
2365 2366 </div> 2367 </div> 2368 </main><footer class="site-footer h-card"> 2369 <data class="u-url" href="/"></data> 2370 2371 <div class="wrapper"> 2372 2373 <div class="footer-col-wrapper"> 2374 <div class="footer-col"> 2375 <!-- <p class="feed-subscribe"> 2376 <a href="/feed.xml"> 2377 <svg class="svg-icon orange"> 2378 <use xlink:href="/assets/minima-social-icons.svg#rss"></use> 2379 </svg><span>Subscribe</span> 2380 </a> 2381 </p> --> 2382 </div> 2383 <div class="footer-col"> 2384 <p style="opacity: 0.2">Thoughts on deep learning, tech, and whatever else seems interesting.</p> 2385 </div> 2386 </div> 2387 2388 <div class="social-links"><style> 2389 .google-scholar-icon { 2390 transform: rotate(180deg); 2391 transform-origin: center; 2392 display: inline-block; 2393 } 2394</style> 2395 2396<ul class="social-media-list"> 2397 <!--<li><a rel="me" href="https://github.com/nikitadurasov" class="svg-a" title="nikitadurasov"><svg class="svg-icon blue"><use xlink:href="/assets/minima-social-icons.svg#github"></use></svg></a></li><li><a rel="me" href="https://www.linkedin.com/in/durasovnikita" class="svg-a" title="durasovnikita"><svg class="svg-icon blue"><use xlink:href="/assets/minima-social-icons.svg#linkedin"></use></svg></a></li><li><a rel="me" href="https://twitter.com/norange_ai" class="svg-a" title="norange_ai"><svg class="svg-icon blue"><use xlink:href="/assets/minima-social-icons.svg#twitter"></use></svg></a></li>--><li><a rel="me" href="https://github.com/nikitadurasov" class="svg-a" title="nikitadurasov"><svg class="svg-icon blue"><use xlink:href="/assets/minima-social-icons.svg#github"></use></svg></a></li><li><a rel="me" href="https://www.linkedin.com/in/durasovnikita" class="svg-a" title="durasovnikita"><svg class="svg-icon blue"><use xlink:href="/assets/minima-social-icons.svg#linkedin"></use></svg></a></li><li><a rel="me" href="https://twitter.com/norange_ai" class="svg-a" title="norange_ai"><svg class="svg-icon blue"><use xlink:href="/assets/minima-social-icons.svg#twitter"></use></svg></a></li><!----><li> 2398 <a rel="me" href="https://scholar.google.com/citations?user=KMMvTfcAAAAJ%26hl" class="svg-a" title="Google Scholar Profile for KMMvTfcAAAAJ&hl"> 2399 <svg class="svg-icon blue google-scholar-icon"> 2400 <use xlink:href="/assets/minima-social-icons.svg#google-scholar"></use> 2401 </svg> 2402 </a> 2403 </li></ul> 2404 2405 2406</div> 2407 2408 </div> 2409 2410</footer> 2411<!-- jQuery first, then Popper.js, then Bootstrap JS. --> 2412
2412<script src="https://code.jquery.com/jquery-3.3.1.slim.min.js" integrity="sha384-q8i/X+965DzO0rT7abK41JStQIAqVgRVzpbzo5smXKp4YfRvH+8abtTE1Pi6jizo" crossorigin="anonymous"></script>
2412 2413
2413<script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.14.7/umd/popper.min.js" integrity="sha384-UO2eT0CpHqdSJQ6hJty5KVphtPhzWj9WO1clHTMGa3JDZwrnQq4sF86dIHNDz0W1" crossorigin="anonymous"></script>
2413 2414
2414<script src="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/js/bootstrap.min.js" integrity="sha384-JjSmVgyd0p3pXB1rRibZUAYoIIy6OrQ6VrjIEaFf/nJGzIxFDsf4x0xIM+B07jRM" crossorigin="anonymous"></script>
2414 2415 2416 </body> 2417 2418</html>
Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.