PageSourceSearch

https://norange.ai/post-training/2026/08/24/post-training-primer.html

html norange.ai collected 2026-10-03 23:20:27 UTC 73,657 bytes, 2,418 lines download raw bytes

1<!DOCTYPE html>
2<html lang="en"><head>
3  <meta charset="utf-8">
4  <meta http-equiv="X-UA-Compatible" content="IE=edge">
5  <meta name="viewport" content="width=device-width, initial-scale=1">
6  
6<script src="https://cdn.jsdelivr.net/npm/chart.js"></script>
6<!-- Begin Jekyll SEO tag v2.8.0 -->
7<title>Post-Training Primer | No-Range Space</title>
8<meta name="generator" content="Jekyll v3.10.0" />
9<meta property="og:title" content="Post-Training Primer" />
10<meta name="author" content="Nikita Durasov" />
11<meta property="og:locale" content="en_US" />
12<meta name="description" content="A first-principles look at post-training for language models, tracing how policy gradients lead to PPO and GRPO. The post breaks down reward-to-go, advantage estimation, probability-ratio clipping, and group-relative rewards, while highlighting the practical differences between PPO’s learned critic and GRPO’s critic-free approach. It also covers implementation details and common pitfalls when putting these methods into practice." />
13<meta property="og:description" content="A first-principles look at post-training for language models, tracing how policy gradients lead to PPO and GRPO. The post breaks down reward-to-go, advantage estimation, probability-ratio clipping, and group-relative rewards, while highlighting the practical differences between PPO’s learned critic and GRPO’s critic-free approach. It also covers implementation details and common pitfalls when putting these methods into practice." />
14<link rel="canonical" href="https://www.norange.ai/post-training/2026/08/24/post-training-primer.html" />
15<meta property="og:url" content="https://www.norange.ai/post-training/2026/08/24/post-training-primer.html" />
16<meta property="og:site_name" content="No-Range Space" />
17<meta property="og:type" content="article" />
18<meta property="article:published_time" content="2026-08-24T00:00:00+00:00" />
19<meta name="twitter:card" content="summary" />
20<meta property="twitter:title" content="Post-Training Primer" />
21<script type="application/ld+json">
22{"@context":"https://schema.org","@type":"BlogPosting","author":{"@type":"Person","name":"Nikita Durasov"},"dateModified":"2026-08-24T00:00:00+00:00","datePublished":"2026-08-24T00:00:00+00:00","description":"A first-principles look at post-training for language models, tracing how policy gradients lead to PPO and GRPO. The post breaks down reward-to-go, advantage estimation, probability-ratio clipping, and group-relative rewards, while highlighting the practical differences between PPO’s learned critic and GRPO’s critic-free approach. It also covers implementation details and common pitfalls when putting these methods into practice.","headline":"Post-Training Primer","mainEntityOfPage":{"@type":"WebPage","@id":"https://www.norange.ai/post-training/2026/08/24/post-training-primer.html"},"url":"https://www.norange.ai/post-training/2026/08/24/post-training-primer.html"}</script>
22
23<!-- End Jekyll SEO tag -->
24<link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro"
25  rel="stylesheet">
26  <link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css" integrity="sha384-ggOyR0iXCbMQv3Xipma34MD+dH/1fQ784/j6cY/iJTQUOhcWr7x9JvoRxT2MZw1T" crossorigin="anonymous">
27  <link rel="stylesheet" href="/assets/css/style.css">
28  <link rel="shortcut icon" type="image/png" href="/assets/images/logo.png">
29  <link href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.0.0/css/all.min.css" rel="stylesheet">
30  <!-- 
30<script async src="https://busuanzi.icodeq.com/busuanzi.pure.mini.js"></script>
30 -->
31  <!-- 
31<script defer src="https://busuanzi.9420.ltd/js"></script>
31 -->
32  <!-- <meta name="referrer" content="no-referrer-when-downgrade"> -->
33  
34  <meta property="og:image" content="/assets/images/logo.png" /><link type="application/atom+xml" rel="alternate" href="https://www.norange.ai/feed.xml" title="No-Range Space" />
vendor: 63 bytes, line 34
34<script async src="https://www.googletagmanager.com/gtag/js?id=
34UA-142853940-1
vendor: 12 bytes, line 34
34"></script>
35<script>
36  window['ga-disable-UA-142853940-1'] = window.doNotTrack === "1" || navigator.doNotTrack === "1" || navigator.doNotTrack === "yes" || navigator.msDoNotTrack === "1";
37  
vendor: 134 bytes, lines 37-41
37window.dataLayer = window.dataLayer || [];
38  function gtag(){dataLayer.push(arguments);}
39  gtag('js', new Date());
40
41  gtag('config', '
41UA-142853940-1
vendor: 4 bytes, line 41
41');
42</script>
42
43<style>
44      .MathJax_Display, .MJXc-display, .MathJax_SVG_Display {
45      overflow-x: auto;
46      overflow-y: hidden;
47  } 
48  </style>
49  <!--
50<meta property="og:image" content="https://www.norange.ai/assets/logo.png" />-->
51</head>
52<body style="font-family: 'Lucida Console', Monaco, monospace"><header class="site-header">
53
54  <div class="wrapper"><div class="row">
55      <div class="col-6 col-sm-3">
56
57        <a rel="author" href="/" class="blog-btn">Blog →</a>
58
59        <style> 
60          .blog-btn { 
61            display: inline-block; 
62            margin: 2px 0;   
63            line-height: 25px;     /* space above & below the button */
64            padding: 3px 10px;    /* space inside the button around the text */
65            font-family: 'Inter', sans-serif; 
66            font-size: 18px; 
67            font-weight: 500; 
68            color: #fff; 
69            text-decoration: none; 
70            border-radius: 8px; 
71            border: 1.5px solid transparent; 
72            background: linear-gradient(#000, #000) padding-box, 
73                        linear-gradient(270deg, #096ce2, #5d41dc) border-box; 
74            transition: all 0.4s ease; 
75            box-shadow: 0 0 12px rgba(131, 56, 236, 0.6); 
76          } 
77
78          .blog-btn:hover { 
79            background: linear-gradient(#000, #000) padding-box, 
80                        linear-gradient(90deg, #FFFFFF, #f285c1) border-box; 
81            box-shadow: 0 0 18px #f285c1; 
82            transform: translateY(-1px); 
83          } 
84
85          .blog-btn:hover::after { 
86            position: relative; 
87            left: 3px; 
88            transition: left 0.0s ease; 
89          }
90
91        </style> 
92
93      </div>
94
95      <div class="col-6 col-sm-9"><nav class="site-nav" style="z-index: 1;">
96            <input type="checkbox" id="nav-trigger" class="nav-trigger" />
97            <label for="nav-trigger">
98              <span class="menu-icon">
99                <svg viewBox="0 0 18 15" width="18px" height="15px">
100                  <path d="M18,1.484c0,0.82-0.665,1.484-1.484,1.484H1.484C0.665,2.969,0,2.304,0,1.484l0,0C0,0.665,0.665,0,1.484,0 h15.032C17.335,0,18,0.665,18,1.484L18,1.484z M18,7.516C18,8.335,17.335,9,16.516,9H1.484C0.665,9,0,8.335,0,7.516l0,0 c0-0.82,0.665-1.484,1.484-1.484h15.032C17.335,6.031,18,6.696,18,7.516L18,7.516z M18,13.516C18,14.335,17.335,15,16.516,15H1.484 C0.665,15,0,14.335,0,13.516l0,0c0-0.82,0.665-1.483,1.484-1.483h15.032C17.335,12.031,18,12.695,18,13.516L18,13.516z"/>
101                </svg>
102              </span>
103            </label>
104
105            <div class="trigger" style="z-index: 1;"><a class="page-link-local" href="/about/">about</a></div>
106          </nav></div>
107    </div>
108
109    <!-- <a class="site-title" rel="author" href="/">No-Range Space</a><nav class="site-nav" style="z-index: 1;">
110        <input type="checkbox" id="nav-trigger" class="nav-trigger" />
111        <label for="nav-trigger">
112          <span class="menu-icon">
113            <svg viewBox="0 0 18 15" width="18px" height="15px">
114              <path d="M18,1.484c0,0.82-0.665,1.484-1.484,1.484H1.484C0.665,2.969,0,2.304,0,1.484l0,0C0,0.665,0.665,0,1.484,0 h15.032C17.335,0,18,0.665,18,1.484L18,1.484z M18,7.516C18,8.335,17.335,9,16.516,9H1.484C0.665,9,0,8.335,0,7.516l0,0 c0-0.82,0.665-1.484,1.484-1.484h15.032C17.335,6.031,18,6.696,18,7.516L18,7.516z M18,13.516C18,14.335,17.335,15,16.516,15H1.484 C0.665,15,0,14.335,0,13.516l0,0c0-0.82,0.665-1.483,1.484-1.483h15.032C17.335,12.031,18,12.695,18,13.516L18,13.516z"/>
115            </svg>
116          </span>
117        </label>
118
119        <div class="trigger" style="z-index: 1;"><a class="page-link-local" href="/about/">about</a></div>
120      </nav>-->
121
122  </div>
123</header>
124<main class="page-content" aria-label="Content">
125      <div class="wrapper">
126        <div style="position: relative; z-index: 0;">
127          
127<script type="text/javascript"
128        src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS_CHTML"></script>
vendor: 1 bytes, line 128
128
129<script src="https://cdn.jsdelivr.net/npm/chart.js"></script>
vendor: 1 bytes, line 129
129
130<script src="https://cdn.jsdelivr.net/npm/[email protected]"></script>
130
131
132<nav class="post-toc" aria-label="Table of contents">
133  <p class="post-toc-title">Contents</p>
134</nav>
135
136<article class="post h-entry" itemscope itemtype="http://schema.org/BlogPosting">
137
138  <header class="post-header">
139    <h1 class="post-title p-name" itemprop="name headline"><span class="post-title-gradient">Post-Training Primer</span></h1> 
140    <p class="post-meta" style="margin: 10px 0px">
141      <span class="title-highlights" style="font-size:15px"><time class="dt-published" datetime="2026-08-24T00:00:00+00:00" itemprop="datePublished">
142        Aug 24, 2026
143      </time>• 
144          <span itemprop="author" itemscope itemtype="http://schema.org/Person">
145            <span class="p-author h-card" itemprop="name">Nikita Durasov</span></span><span style="margin-left: 8px;">•</span> <i class="fa-solid fa-eye"></i>
146      <span><span id="ga_site_pv">Loading...</span> views</span>
147      </span>
148    </p>
149  </header>
150
151  <div class="post-content e-content" itemprop="articleBody">
152    <style>
153.reference {
154  color: #7c3aed;
155  font-weight: 650;
156}
157.readable-code pre,
158.readable-code code {
159  color: #e5e7eb !important;
160  background: #111827 !important;
161}
162.readable-code pre {
163  border: 1px solid #374151;
164  border-radius: 10px;
165  padding: 1rem 1.1rem;
166}
167</style>
168
169<h2 id="introduction">Introduction</h2>
170
171<h3 id="post-training-unified">Post-Training Unified</h3>
172
173<p>Pretraining teaches a model to represent and continue the distribution of its training data. Post-training changes that distribution toward behavior we care about: following instructions, solving tasks, respecting preferences, using tools, or maximizing a measurable reward.</p>
174
175<p>For language models, the object being trained is still a conditional probability distribution,</p>
176
177\[\pi_{\theta}(o_t \mid q, o_{&lt;t}),\]
178
179<p>where \(q\) is a prompt, \(o_{&lt;t}\) is the response prefix, and \(o_t\) is the next token. What changes between supervised fine-tuning, PPO, and GRPO is primarily the coefficient placed in front of the familiar log-likelihood gradient</p>
180
181\[\nabla_{\theta}\log \pi_{\theta}(o_t \mid q,o_{&lt;t}).\]
182
183<p>This gives a useful unifying picture:</p>
184
185<ul>
186  <li>
187    <p>Supervised fine-tuning increases the probability of selected demonstrations.</p>
188  </li>
189  <li>
190    <p>Policy gradients increase or decrease the probability of sampled actions according to their return.</p>
191  </li>
192  <li>
193    <p>PPO uses an advantage estimate and constrains how far the new policy can move from the policy that generated the data.</p>
194  </li>
195  <li>
196    <p>GRPO retains the PPO update but estimates the baseline by comparing several responses to the same prompt instead of training a critic.</p>
197  </li>
198</ul>
199
200<p>The differences between these methods then come down to a more subtle question: what signal should multiply the log-likelihood gradient, and why? That question leads naturally from likelihood to return, from return to advantage, and from advantage to the particular updates used by PPO and GRPO.</p>
201
202<h3 id="reinforcement-learning-notation">Reinforcement Learning Notation</h3>
203
204<p>Consider an episodic Markov decision process. At step \(t\), an agent observes a state \(s_t\), samples an action \(a_t\), receives a reward \(r_t\), and moves to \(s_{t+1}\). A trajectory is</p>
205
206\[\tau=(s_0,a_0,r_0,s_1,a_1,r_1,\ldots,s_T).\]
207
208<p>The policy is a probability distribution</p>
209
210\[\pi_{\theta}(a_t\mid s_t),\]
211
212<p>parameterized by \(\theta\). For now, define the undiscounted trajectory return as</p>
213
214\[R(\tau)=\sum_{t=0}^{T-1}r_t.\]
215
216<p>The training objective is</p>
217
218\[J(\theta)=\mathbb{E}_{\tau\sim\pi_{\theta}}[R(\tau)].\]
219
220<p>For a language model, the translation is direct:</p>
221
222\[s_t=(q,o_{&lt;t}),\qquad a_t=o_t.\]
223
224<p>The environment state is the prompt and generated prefix, and the action is the next token. The probability of a complete response factorizes autoregressively:</p>
225
226\[\pi_{\theta}(o\mid q)
227
228=
229
230\prod_{t=1}^{|o|}\pi_{\theta}(o_t\mid q,o_{&lt;t}).\]
231
232<h2 id="policy-gradient">Policy Gradient</h2>
233
234<h3 id="reinforce">REINFORCE</h3>
235
236<p>The probability of a trajectory is</p>
237
238\[p_{\theta}(\tau)
239
240=
241
242p(s_0)
243
244\prod_{t=0}^{T-1}
245
246\pi_{\theta}(a_t\mid s_t)
247
248P(s_{t+1}\mid s_t,a_t).\]
249
250<p>The initial-state distribution and environment dynamics do not depend on the policy parameters. Starting from the objective,</p>
251
252\[J(\theta)=\sum_{\tau}p_{\theta}(\tau)R(\tau),\]
253
254<p>we differentiate:</p>
255
256\[\nabla_{\theta}J(\theta)
257
258=
259
260\sum_{\tau}\nabla_{\theta}p_{\theta}(\tau)R(\tau).\]
261
262<p>Use the log-derivative identity</p>
263
264<details class="margin-note">
265  <summary>What is log-derivative trick?</summary>
266
267  Start with the derivative of the logarithm:
268
269  $$
270
271  \nabla_{\theta}\log p_{\theta}(\tau)
272  =
273  \frac{1}{p_{\theta}(\tau)}
274  \nabla_{\theta}p_{\theta}(\tau)
275
276  $$
277
278  Multiplying both sides by \(p_{\theta}(\tau)\) gives:
279
280  $$
281
282  \nabla_{\theta}p_{\theta}(\tau)
283  =
284  p_{\theta}(\tau)
285  \nabla_{\theta}\log p_{\theta}(\tau)
286  
287  $$
288
289  This is the <strong>log-derivative trick</strong>, also called the <strong>score-function identity</strong>.
290</details>
291
292\[\nabla_{\theta}p_{\theta}(\tau)
293
294=
295
296p_{\theta}(\tau)\nabla_{\theta}\log p_{\theta}(\tau).\]
297
298<p>Then</p>
299
300\[\nabla_{\theta}J(\theta)
301
302=
303
304\mathbb{E}_{\tau\sim\pi_{\theta}}
305
306\left[
307
308R(\tau)\nabla_{\theta}\log p_{\theta}(\tau)
309
310\right].\]
311
312<p>Because only the policy factors depend on \(\theta\),</p>
313
314\[\nabla_{\theta}\log p_{\theta}(\tau)
315
316=
317
318\sum_{t=0}^{T-1}
319
320\color{#2563eb}{\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t)}.\]
321
322<p>Therefore,</p>
323
324\[\nabla_{\theta}J(\theta)
325
326=
327
328\mathbb{E}
329
330\left[
331
332R(\tau)
333
334\sum_{t=0}^{T-1}
335
336\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t)
337
338\right].\]
339
340<p>This is the score-function or REINFORCE estimator <span class="reference">[Wil92]</span>. It is the central mechanism behind the methods discussed below. We do not differentiate through the sampled discrete action. Instead, we differentiate the log-probability of having sampled it.</p>
341
342<h3 id="why-past-rewards-disappear">Why Past Rewards Disappear</h3>
343
344<p>The previous formula multiplies every policy-score term by the entire trajectory return. At first this seems to imply that an action at time \(10\) should be credited for a reward received at time \(0\). Causally, that cannot be right: the later action cannot change an event that has already happened.</p>
345
346<p>The mathematical statement is exactly</p>
347
348\[\mathbb{E}
349
350\left[
351
352 r_0\nabla_{\theta}\log\pi_{\theta}(a_{10}\mid s_{10})
353
354\right]
355
356=0.\]
357
358<p>More generally, for every \(k\lt t\),</p>
359
360\[\mathbb{E}
361
362\left[
363
364 r_k\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t)
365
366\right]
367
368=0.\]
369
370<h3 id="conditional-expectation-proof">Conditional-expectation proof</h3>
371
372<p>Let \(H_t\) denote everything known immediately before sampling \(a_t\):</p>
373
374\[H_t=(s_0,a_0,r_0,\ldots,s_t).\]
375
376<p>For \(k\lt t\), the reward \(r_k\) is already contained in \(H_t\). It is therefore fixed after conditioning on \(H_t\). By the law of total expectation,</p>
377
378\[\begin{aligned}
379
380&amp;\mathbb{E}
381
382\left[
383
384 r_k\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t)
385
386\right]
387
388\\\\
389
390&amp;=\mathbb{E}
391
392\left[
393
394\mathbb{E}
395
396\left[
397
398 r_k\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t)
399
400\mid H_t
401
402\right]
403
404\right]
405
406\\\\
407
408&amp;=\mathbb{E}
409
410\left[
411
412 r_k
413
414\mathbb{E}
415
416\left[
417
418\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t)
419
420\mid H_t
421
422\right]
423
424\right].
425
426\end{aligned}\]
427
428<p>Conditional on the history, the only remaining randomness in this expression is the newly sampled action</p>
429
430\[a_t\sim\pi_{\theta}(\cdot\mid s_t).\]
431
432<p>The expected score of a normalized probability distribution is zero:</p>
433
434\[\begin{aligned}
435
436\mathbb{E}_{a_t\sim\pi_{\theta}}
437
438\left[
439
440\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t)
441
442\right]
443
444&amp;=
445
446\sum_a
447
448\pi_{\theta}(a\mid s_t)
449
450\nabla_{\theta}\log\pi_{\theta}(a\mid s_t)
451
452\\\\
453
454&amp;=
455
456\sum_a\nabla_{\theta}\pi_{\theta}(a\mid s_t)
457
458\\\\
459
460&amp;=
461
462\nabla_{\theta}\sum_a\pi_{\theta}(a\mid s_t)
463
464\\\\
465
466&amp;=
467
468\nabla_{\theta}1
469
470=0.
471
472\end{aligned}\]
473
474<p>Substituting this into the conditional expectation proves the claim.</p>
475
476<p>A subtle point is worth emphasizing. The past reward \(r_k\) can be highly correlated with the later state \(s_t\). For example, both may depend on earlier actions. Independence is not required. The argument works because, after conditioning on the history, the expected score of the newly sampled action is zero.</p>
477
478<p>The statement concerns</p>
479
480\[\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t),\]
481
482<p>not the gradient of a joint marginal such as \(\nabla_{\theta}\log p_{\theta}(a_t,s_t)\). The distribution of \(s_t\) depends on previous policy decisions, so the latter contains additional terms.</p>
483
484<h3 id="from-the-proof-to-reward-to-go">From the proof to reward-to-go</h3>
485
486<p>Write</p>
487
488\[g_t=\color{#2563eb}{\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t)}.\]
489
490<p>Then</p>
491
492\[\begin{aligned}
493
494\nabla_{\theta}J
495
496&amp;=
497
498\sum_t
499
500\mathbb{E}
501
502\left[
503
504\left(\sum_k r_k\right)g_t
505
506\right]
507
508\\\\
509
510&amp;=
511
512\sum_t\sum_k\mathbb{E}[r_k g_t].
513
514\end{aligned}\]
515
516<p>Every term with \(k\lt t\) is zero. Consequently,</p>
517
518\[\begin{aligned}
519
520\nabla_{\theta}J
521
522&amp;=
523
524\sum_t
525
526\mathbb{E}
527
528\left[
529
530\left(\sum_{k=t}^{T-1}
530r_k\right)g_t
531
532\right]
533
534\\\\
535
536&amp;=
537
538\mathbb{E}
539
540\left[
541
542\sum_t G_t
543
544\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t)
545
546\right],
547
548\end{aligned}\]
549
550<p>where</p>
551
552\[\boxed{
553
554\color{#16a34a}{G_t}=\sum_{k=t}^{T-1}r_k
555
556}\]
557
558<p>is the reward-to-go for the undiscounted objective. If the objective is</p>
559
560\[J_{\gamma}(\theta)
561
562=
563
564\mathbb{E}
565
566\left[
567
568\sum_{k=0}^{T-1}\gamma^k r_k
569
570\right],\]
571
572<p>define the locally discounted reward-to-go</p>
573
574<details class="margin-note"><summary>Why introduce discounting?</summary>Discounting makes future rewards count less than immediate rewards. This is useful when we care more about near-term outcomes, when delayed rewards are less certain, or when we want to limit the effective planning horizon. 
575<br /><br />
576It can also reduce the variance of policy-gradient estimates by downweighting distant, noisy rewards, although this comes at the cost of changing the objective: the discounted and undiscounted problems are not equivalent.</details>
577
578\[G_t^{(\gamma)}
579
580=
581
582\sum_{\ell=0}^{T-t-1}\gamma^{\ell}r_{t+\ell}.\]
583
584<p>The exact episodic gradient then contains the discount accumulated before time \(t\):</p>
585
586\[\nabla_{\theta}J_{\gamma}
587
588=
589
590\mathbb{E}
591
592\left[
593
594\sum_t
595
596\gamma^t G_t^{(\gamma)}
597
598\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t)
599
600\right].\]
601
602<p>Some presentations absorb this outer factor into the discounted state-visitation distribution, so it may not appear explicitly. The convention should be stated rather than silently mixed.</p>
603
604<p>Reward-to-go is therefore not merely a heuristic. It is obtained by deleting terms whose expectation is exactly zero. This normally reduces variance because the score for \(a_t\) is no longer multiplied by irrelevant random rewards from the past.</p>
605
606<h2 id="baselines-and-advantages">Baselines and Advantages</h2>
607
608<p>Throughout this section, expectations are taken over trajectories generated by the current policy \(\pi_\theta\) and the environment dynamics. When we condition on \((s_t)\) or \((s_t,a_t)\), the expectation is instead over the corresponding conditional distribution of future trajectory outcomes. In particular, (G_t) is a sampled return from a trajectory, while (V^\pi) and (Q^\pi) are conditional expectations of that return.</p>
609
610<p>The same zero-score identity lets us subtract a baseline without changing the expected policy gradient. Let (b(H_t)) be any quantity known before sampling (a_t). Then</p>
611
612\[\underset{
613\substack{
614s_t\sim d^\pi_t\\
615a_t\sim\pi_\theta(\cdot\mid s_t)
616}
617}{
618\mathbb{E}
619}
620\left[
621b(H_t)
622\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t)
623\right]
624
625=0.\]
626
627<p>Thus,</p>
628
629\[\boxed{
630
631\nabla_{\theta}J
632
633=
634
635\mathbb{E}_{\tau\sim p_\theta}
636
637\left[
638
639\sum_t
640
641\left(G_t-b(H_t)\right)
642
643\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t)
644
645\right].
646
647}\]
648
649<p>The baseline does not need to be constant. It can depend on the state or the full pre-action history. It must not depend on the current sampled action in a way that invalidates the zero-score argument.</p>
650
651<p>If a learned baseline shares parameters with the policy, its value is normally detached in the actor loss. Otherwise, automatic differentiation adds a gradient through the baseline itself, which is not part of the policy-gradient identity above.</p>
652
653<p>The most important baseline is the value function</p>
654
655\[V^{\pi}(s)
656=
657\mathbb{E}_{\tau_{t:T}\sim p^\pi(\tau_{t:T}\mid s_t=s)}
658\left[
659G_t
660\right].\]
661
662<p>It is the expected future return when the agent is in state (s) and subsequently follows policy (\pi).</p>
663
664<p>Define the action-value function</p>
665
666\[Q^{\pi}(s,a)
667=
668\mathbb{E}_{\tau_{t:T}\sim p^\pi(\tau_{t:T}\mid s_t=s,a_t=a)}
669\left[
670G_t
671\right]\]
672
673<p>and the advantage</p>
674
675\[\boxed{
676\color{#16a34a}{A^{\pi}(s,a)}
677=
678Q^{\pi}(s,a)-V^{\pi}(s).
679}\]
680
681<p>The advantage asks a relative question: was this action better or worse than what the policy usually does in this state?</p>
682
683<ul>
684  <li>
685    <p>(A&gt;0): increase the action probability.</p>
686  </li>
687  <li>
688    <p>(A&lt;0): decrease the action probability.</p>
689  </li>
690  <li>
691    <p>(A=0): this sample supplies no first-order preference.</p>
692  </li>
693</ul>
694
695<p>The policy-gradient theorem can therefore be written in its familiar form <sp
695an class="reference">[Sut99]</span>:</p>
696
697\[\nabla_{\theta}J
698=
699\underset{
700\substack{
701s_t\sim d^\pi_t\\
702a_t\sim\pi_\theta(\cdot\mid s_t)
703}
704}{
705\mathbb{E}
706}
707\left[
708
709A^{\pi}(s_t,a_t)
710
711\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t)
712
713\right].\]
714
715<h2 id="the-critic-and-generalized-advantage-estimation">The Critic and Generalized Advantage Estimation</h2>
716
717<p>The policy gradient tells us that actions with higher-than-expected returns should become more likely, while actions with lower-than-expected returns should become less likely.</p>
718
719<p>To do this, we need an estimate of the <strong>advantage</strong> of the action that was actually taken.</p>
720
721<p>The true value function is</p>
722
723\[V^\pi(s)
724=
725\mathbb{E}[G_t\mid s_t=s].\]
726
727<p>It tells us the expected return when starting from state (s) and following the current policy.</p>
728
729<p>In practice, we do not know the true value function. PPO therefore learns a parameterized approximation, called the <strong>critic</strong>:</p>
730
731\[V_\psi(s)
732\approx
733V^\pi(s).\]
734
735<p>The critic is usually a neural network with parameters (\psi). Its job is not to choose actions, but to predict how much return we can expect from a state.</p>
736
737<h3 id="monte-carlo-advantage-estimate">Monte Carlo advantage estimate</h3>
738
739<p>Suppose we have collected a trajectory and observed the return from time (t) onward:</p>
740
741\[G_t
742=
743r_t
744+
745\gamma r_{t+1}
746+
747\gamma^2 r_{t+2}
748+\cdots.\]
749
750<p>A simple estimate of the advantage is then</p>
751
752\[\hat A_t
753=
754G_t
755-
756V_\psi(s_t).\]
757
758<p>The intuition is straightforward:</p>
759
760<ul>
761  <li>if the observed return is higher than the critic expected, the action performed better than expected;</li>
762  <li>if the observed return is lower than expected, the action performed worse than expected.</li>
763</ul>
764
765<p>However, the Monte Carlo return uses the entire future trajectory. This can make the estimate noisy, especially for long episodes.</p>
766
767<h3 id="one-step-temporal-difference-error">One-step temporal-difference error</h3>
768
769<p>Instead of waiting until the end of the episode, we can use the critic to estimate the future.</p>
770
771<p>Define the <strong>temporal-difference residual</strong> as</p>
772
773\[\delta_t
774=
775r_t
776+
777\gamma V_\psi(s_{t+1})
778-
779V_\psi(s_t).\]
780
781<p>This compares two quantities.</p>
782
783<p>The critic predicted</p>
784
785\[V_\psi(s_t).\]
786
787<p>After taking the action, we observe the immediate reward and use the critic’s prediction at the next state:</p>
788
789\[r_t+\gamma V_\psi(s_{t+1}).\]
790
791<p>The difference is therefore</p>
792
793\[\delta_t
794=
795\text{new estimate}
796-
797\text{old estimate}.\]
798
799<p>It can be interpreted as a <strong>prediction error</strong>: the outcome was better or worse than the critic expected.</p>
800
801<p>If the critic were perfect, then</p>
802
803\[V_\psi(s)
804=
805V^\pi(s).\]
806
807<p>In that case, conditioning on the current state and action gives</p>
808
809\[\mathbb{E}
810\left[
811\delta_t
812\mid
813s_t=s,\,
814a_t=a
815\right]
816=
817A^\pi(s,a).\]
818
819<p>To see why, recall that the action-value function is</p>
820
821\[Q^\pi(s,a)
822=
823\mathbb{E}
824\left[
825r_t+\gamma V^\pi(s_{t+1})
826\mid
827s_t=s,\,
828a_t=a
829\right].\]
830
831<p>Therefore,</p>
832
833\[\begin{aligned}
834\mathbb{E}
835\left[
836\delta_t
837\mid
838s_t=s,\,
839a_t=a
840\right]
841&amp;=
842Q^\pi(s,a)-V^\pi(s)
843\\
844&amp;=
845A^\pi(s,a).
846\end{aligned}\]
847
848<p>So a one-step TD residual already provides an advantage estimate when the critic is exact.</p>
849
850<p>The problem is that in practice the critic is not exact. We can therefore look further into the future and combine several TD residuals.</p>
851
852<h3 id="multi-step-advantage-estimates">Multi-step advantage estimates</h3>
853
854<p>Consider a sum of (k) consecutive TD residuals:</p>
855
856\[\hat A_t^{(k)}
857=
858\sum_{\ell=0}^{k-1}
859\gamma^\ell
860\delta_{t+\ell}.\]
861
862<p>Substituting the definition of the TD residual gives</p>
863
864\[\hat A_t^{(k)}
865=
866\sum_{\ell=0}^{k-1}
867\gamma^\ell
868\left(
869r_{t+\ell}
870+
871\gamma V_\psi(s_{t+\ell+1})
872-
873V_\psi(s_{t+\ell})
874\right).\]
875
876<p>The value-function terms in the middle cancel each other. What remains is</p>
877
878\[\hat A_t^{(k)}
879=
880-
881V_\psi(s_t)
882+
883\sum_{\ell=0}^{k-1}
884\gamma^\ell r_{t+\ell}
885+
886\gamma^k V_\psi(s_{t+k}).\]
887
888<p>
888This is a (k)-step return minus the value estimate at the starting state.</p>
889
890<p>For (k=1), we recover the one-step TD residual:</p>
891
892\[\hat A_t^{(1)}
893=
894\delta_t.\]
895
896<p>At the other extreme, suppose the trajectory terminates after the final reward and the terminal value is zero:</p>
897
898\[V_\psi(s_T)=0.\]
899
900<p>Taking the full remaining trajectory gives</p>
901
902\[\hat A_t^{(T-t)}
903=
904G_t-V_\psi(s_t).\]
905
906<p>Thus, multi-step estimates provide a spectrum between one-step bootstrapping and a full Monte Carlo return.</p>
907
908<h3 id="generalized-advantage-estimation">Generalized Advantage Estimation</h3>
909
910<p><strong>Generalized Advantage Estimation</strong>, or <strong>GAE</strong>, combines TD residuals over multiple future time steps:</p>
911
912\[\boxed{
913\hat A_t^{\mathrm{GAE}(\gamma,\lambda)}
914=
915\sum_{\ell=0}^{T-t-1}
916(\gamma\lambda)^\ell
917\delta_{t+\ell}
918}\]
919
920<p>The term</p>
921
922\[(\gamma\lambda)^\ell\]
923
924<p>controls how much influence a TD residual has as it gets further away from the current time step.</p>
925
926<p>Expanding the first few terms makes the idea clearer:</p>
927
928\[\hat A_t
929=
930\delta_t
931+
932\gamma\lambda\delta_{t+1}
933+
934(\gamma\lambda)^2\delta_{t+2}
935+
936\cdots.\]
937
938<p>The parameter (\lambda) therefore controls how far into the future we use TD errors.</p>
939
940<p>When</p>
941
942\[\lambda=0,\]
943
944<p>GAE becomes</p>
945
946\[\hat A_t
947=
948\delta_t.\]
949
950<p>We only use the immediate TD residual. This generally gives lower variance, but it relies more heavily on the critic.</p>
951
952<p>When</p>
953
954\[\lambda=1,\]
955
956<p>we obtain</p>
957
958\[\hat A_t
959=
960\sum_{\ell=0}^{T-t-1}
961\gamma^\ell\delta_{t+\ell}.\]
962
963<p>For a terminated trajectory, the TD residuals telescope, giving</p>
964
965\[\hat A_t
966=
967G_t-V_\psi(s_t).\]
968
969<p>Thus, the two extremes are approximately:</p>
970
971\[\lambda=0
972\quad\Longrightarrow\quad
973\text{one-step TD}\]
974
975<p>and</p>
976
977\[\lambda=1
978\quad\Longrightarrow\quad
979\text{Monte Carlo}.\]
980
981<p>Intermediate values of (\lambda) provide a compromise between these two extremes.</p>
982
983<p>A smaller value of (\lambda) puts more weight on short-term TD estimates. This reduces variance but can increase bias when the critic is inaccurate.</p>
984
985<p>A larger value of (\lambda) incorporates more of the actual future trajectory. This reduces reliance on the critic, but generally increases variance.</p>
986
987<h3 id="computing-gae-backwards">Computing GAE backwards</h3>
988
989<p>GAE is usually computed efficiently by working backwards through the trajectory.</p>
990
991<p>Let (d_t) indicate whether the transition at time (t) terminates the episode:</p>
992
993\[d_t
994=
995\begin{cases}
9961, &amp; \text{if the transition terminates the episode},\\
9970, &amp; \text{otherwise}.
998\end{cases}\]
999
1000<p>The recursive form is</p>
1001
1002\[\boxed{
1003\hat A_t
1004=
1005\delta_t
1006+
1007\gamma\lambda
1008(1-d_t)
1009\hat A_{t+1}
1010}\]
1011
1012<p>where</p>
1013
1014\[\delta_t
1015=
1016r_t
1017+
1018\gamma(1-d_t)
1019V_\psi(s_{t+1})
1020-
1021V_\psi(s_t).\]
1022
1023<p>The termination mask prevents information from one episode from leaking into the next.</p>
1024
1025<p>This backward recursion is the form commonly used in PPO implementations.</p>
1026
1027<h3 id="actor-and-critic-together">Actor and critic together</h3>
1028
1029<p>The actor and critic therefore have different jobs.</p>
1030
1031<p>The <strong>actor</strong> represents the policy:</p>
1032
1033\[\pi_\theta(a_t\mid s_t).\]
1034
1035<p>It determines which actions are taken and is updated using the estimated advantages.</p>
1036
1037<p>The <strong>critic</strong> represents the value estimate:</p>
1038
1039\[V_\psi(s_t).\]
1040
1041<p>It predicts the expected return and provides the baseline needed to construct lower-variance advantage estimates.</p>
1042
1043<p>During PPO training, the two networks are updated together but optimize different objectives: the actor improves the policy, while the critic improves its prediction of future returns.</p>
1044
1045<p>The overall picture is therefore:</p>
1046
1047\[\boxed{
1048\text{trajectory}
1049\;\longrightarrow\;
1050\text{TD residuals}
1051\;\longrightarrow\;
1052\text{GAE advantages}
1053\;\longrightarrow\;
1054\text{policy update}
1055}\]
1056
1057<p>while the critic is simultaneously trained to become a better estimate of the value function.</p>
1058
1059<h2 id="probability-ratios">Probability Ratios</h2>
1060
1061<p>Suppose trajectories were sampled from an old policy \(\pi_{\mathrm{old}}\), but we are now optimizing a new policy \(\pi_{\theta}\). At a fixed state,</p>
1062
1063\[\mathbb{E}_{a\sim\pi_{\theta}}[A_{\mathrm{old}}(s,a)]
1064
1065=
1066
1067\sum_a\pi_{\theta}(a\mid s)A_{\mathrm{old}}(s,a).\]
1068
1069<p>Insert the old policy:</p>
1070
1071\[\begin{aligned}
1072
1073\sum_a\pi_{\theta}(a\mid s)A_{\mathrm{old}}(s,a)
1074
1075&amp;=
1076
1077\sum_a
1078
1079\pi_{\mathrm{old}}(a\mid s)
1080
1081\frac{\pi_{\theta}(a\mid s)}
1082
1083{\pi_{\mathrm{old}}(a\mid s)}
1084
1085A_{\mathrm{old}}(s,a).
1086
1087\end{aligned}\]
1088
1089<p>Define the importance ratio</p>
1090
1091\[\boxed{
1092
1093\color{#2563eb}{\rho_t(\theta)}
1094
1095=
1096
1097\frac{\pi_{\theta}(a_t\mid s_t)}
1098
1099{\pi_{\mathrm{old}}(a_t\mid s_t)}.
1100
1101}\]
1102
1103<p>This produces the surrogate objective</p>
1104
1105\[L^{\mathrm{CPI}}(\theta)
1106
1107=
1108
1109\mathbb{E}_{t\sim\pi_{\mathrm{old}}}
1110
1111[\color{#2563eb}{\rho_t(\theta)}\hat A_t].\]
1112
1113<p>At the start of an update, when \(\theta=\theta_{\mathrm{old}}\), we have \(\rho_t=1\). Moreover,</p>
1114
1115\[\nabla_{\theta}\rho_t
1116
1117=
1118
1119\rho_t\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t),\]
1120
1121<p>so the gradient of the surrogate at the old policy is the ordinary advantage-weighted policy gradient.</p>
1122
1123<p>The ratio corrects the action distribution at states sampled by the old policy. It does not fully correct the state-visitation distribution, which also changes when the policy changes. This is why the surrogate is most reliable for a local policy update. TRPO formalizes this idea with a KL-divergence trust region <span class="reference">[Sch15a]</span>. PPO replaces the constrained second-order update with a simpler clipped first-order objective <span class="reference">[Sch17]</span>.</p>
1124
1125<h2 id="clipped-objective">Clipped Objective</h2>
1126
1127<p>PPO maximizes</p>
1128
1129\[\boxed{
1130
1131L^{\mathrm{CLIP}}(\theta)
1132
1133=
1134
1135\mathbb{E}_t
1136
1137\left[
1138
1139\min
1140
1141\left(
1142
1143\rho_t\hat A_t,
1144
1145\operatorname{clip}(\rho_t,1-\epsilon,1+\epsilon)\hat A_t
1146
1147\right)
1148
1149\right].
1150
1151}\]
1152
1153<p>
1153Optimizers usually minimize a loss, so implementations use</p>
1154
1155\[\mathcal{L}_{\mathrm{policy}}
1156
1157=-L^{\mathrm{CLIP}}.\]
1158
1159<p>The minimum makes the objective pessimistic: once a sampled action has moved far enough in the favorable direction, that sample stops rewarding an even larger move.</p>
1160
1161<h3 id="positive-advantage">Positive advantage</h3>
1162
1163<p>When \(\hat A_t&gt;0\), increasing the action probability is favorable. The objective becomes</p>
1164
1165\[\ell_t
1166
1167=
1168
1169\hat A_t\min(\rho_t,1+\epsilon).\]
1170
1171<p>For \(\rho_t&gt;1+\epsilon\), the sample is saturated and supplies no further policy gradient.</p>
1172
1173<h3 id="negative-advantage">Negative advantage</h3>
1174
1175<p>When \(\hat A_t&lt;0\), decreasing the action probability is favorable. Because multiplication by a negative value reverses the ordering,</p>
1176
1177\[\ell_t
1178
1179=
1180
1181\hat A_t\max(\rho_t,1-\epsilon).\]
1182
1183<p>For \(\rho_t&lt;1-\epsilon\), the sample is saturated and supplies no further policy gradient.</p>
1184
1185<p>Clipping is asymmetric in an important way: it suppresses an excessively large favorable move, not an unfavorable move. If a positive-advantage action becomes less likely, PPO leaves the corrective gradient active.</p>
1186
1187<h2 id="gradient-of-the-ppo-loss">Gradient of the PPO Loss</h2>
1188
1189<p>Ignoring the exact nondifferentiable clipping boundaries, define</p>
1190
1191\[m_t
1192
1193=
1194
1195\mathbf{1}[\hat A_t&gt;0,\rho_t&lt;1+\epsilon]
1196
1197+
1198
1199\mathbf{1}[\hat A_t&lt;0,\rho_t&gt;1-\epsilon].\]
1200
1201<p>Then</p>
1202
1203\[\boxed{
1204
1205\nabla_{\theta}L^{\mathrm{CLIP}}
1206
1207=
1208
1209\mathbb{E}_t
1210
1211\left[
1212
1213 m_t\rho_t\hat A_t
1214
1215\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t)
1216
1217\right].
1218
1219}\]
1220
1221<p>The minimized policy loss has the opposite sign:</p>
1222
1223\[\boxed{
1224
1225\nabla_{\theta}\mathcal{L}_{\mathrm{policy}}
1226
1227=-
1228
1229\mathbb{E}_t
1230
1231\left[
1232
1233 m_t\rho_t\hat A_t
1234
1235\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t)
1236
1237\right].
1238
1239}\]
1240
1241<p>This is still advantage-weighted log-likelihood. PPO only changes the coefficient by adding the old-to-new ratio and the clipping mask.</p>
1242
1243<h3 id="gradient-with-respect-to-logits">Gradient with respect to logits</h3>
1244
1245<p>Suppose the policy produces logits \(z_j\) and probabilities</p>
1246
1247\[\pi_j=\frac{e^{z_j}}{\sum_k e^{z_k}}.\]
1248
1249<p>For the sampled action \(a\),</p>
1250
1251\[\frac{\partial\log\pi(a\mid s)}{\partial z_j}
1252
1253=
1254
1255\mathbf{1}[j=a]-\pi_j.\]
1256
1257<p>In an active, unclipped region,</p>
1258
1259\[\frac{\partial\mathcal{L}_{\mathrm{policy},t}}{\partial z_j}
1260
1261=
1262
1263-\rho_t\hat A_t
1264
1265\left(\mathbf{1}[j=a_t]-\pi_j\right).\]
1266
1267<p>For positive advantage, gradient descent raises the selected action’s logit relative to the alternatives. For negative advantage, it lowers it.</p>
1268
1269<h2 id="the-full-ppo-loss">The Full PPO Loss</h2>
1270
1271<p>A common minimized PPO objective is</p>
1272
1273\[\boxed{
1274
1275\color{#7c3aed}{\mathcal{L}_{\mathrm{PPO}}}
1276
1277=
1278
1279\mathcal{L}_{\mathrm{policy}}
1280
1281+c_V\mathcal{L}_{\mathrm{value}}
1282
1283-c_H\mathcal{H}.
1284
1285}\]
1286
1287<p>The value loss is often</p>
1288
1289\[\mathcal{L}_{\mathrm{value}}
1290
1291=
1292
1293\frac{1}{2}
1294
1295\mathbb{E}_t
1296
1297\left[
1298
1299\left(V_{\psi}(s_t)-\hat R_t\right)^2
1300
1301\right],\]
1302
1303<p>where a common target is</p>
1304
1305\[\hat R_t=\hat A_t+V_{\mathrm{old}}(s_t).\]
1306
1307<p>The entropy term</p>
1308
1309\[\mathcal{H}
1310
1311=
1312
1313\mathbb{E}_s
1314
1315\left[
1316
1317-\sum_a\pi_{\theta}(a\mid s)
1318
1319\log\pi_{\theta}(a\mid s)
1320
1321\right]\]
1322
1323<p>encourages exploration when it is subtracted from the minimized loss.</p>
1324
1325<p>During the actor update, the old log-probabilities, returns, and advantages are treated as constants. The critic is trained through its separate value loss.</p>
1326
1327<h2 id="ppo-for-language-models">PPO for Language Models</h2>
1328
1329<p>For an autoregressive language model,</p>
1330
1331\[s_t=(q,o_{&lt;t}),\qquad a_t=o_t.\]
1332
1333<p>The token-level PPO ratio is</p>
1334
1335\[\rho_t
1336
1337=
1338
1339\frac{
1340
1341\pi_{\theta}(o_t\mid q,o_{&lt;t})
1342
1343}{
1344
1345\pi_{\mathrm{old}}(o_t\mid q,o_{&lt;t})
1346
1347}.\]
1348
1349<p>It is evaluated stably in log-space:</p>
1350
1351\[\rho_t
1352
1353=
1354
1355\exp
1356
1357\left(
1358
1359\log\pi_{\theta}(o_t\mid q,o_{&lt;t})
1360
1361-
1362
1363\log\pi_{\mathrm{old}}(o_t\mid q,o_{&lt;t})
1364
1365\right).\]
1366
1367<p>A sequence-level ratio would multiply all token ratios. For long responses this product can become extremely large or small. Token-level PPO instead views every prefix as a state and every next token as an action.</p>
1368
1369<p>Three models must be kept conceptually separate:</p>
1370
1371<p>1. The current policy \(\pi_{\theta}\) is being optimized.</p>
1372
1373<p>2. The old policy \(\pi_{\mathrm{old}}\) generated the current rollouts and appears in the PPO ratio.</p>
1374
1375<p>3. The reference policy \(\pi_{\mathrm{ref}}\) regularizes behavior toward a stable language model, often the supervised fine-tuned model.</p>
1376
1377<p>PPO also normally uses a fourth component, the critic \(V_{\psi}\). The old policy and reference policy may initially contain identical weights, but they have different mathematical roles. The old policy is part of importance weighting; the reference policy is part of behavioral regularization.</p>
1378
1379<p>For language-model RL, the task reward often arrives only after the complete answer. The critic attempts to predict the expected final outcome from each partial response. This can provide token-specific credit assignment, but it requires training and storing an additional value model. GRPO removes that component.</p>
1380
1381<h2 id="group-relative-policy-optimization">Group Relative Policy Optimization</h2>
1382
1383<p>GRPO was introduced as a critic-free variant of PPO in DeepSeekMath <span class="reference">[Sha24]</span>. For each prompt \(q\), sample a group of \(G\) responses from the old policy:</p>
1384
1385\[o_1,\ldots,o_G
1386
1387\sim
1388
1389\pi_{\mathrm{old}}(\cdot\mid q).\]
1390
1391<p>Score them with a reward function:</p>
1392
1393\[R_1,\ldots,R_G.\]
1394
1395<p>Compute prompt-local statistics</p>
1396
1397\[\mu_q
1398
1399=
1400
1401\frac{1}{G}\sum_{i=1}^G R_i,\]
1402
1403\[\sigma_q
1404
1405=
1406
1407\sqrt{
1408
1409\frac{1}{G}\sum_{i=1}^G(R_i-\mu_q)^2
1410
1411+\varepsilon_{\mathrm{std}}
1412
1413},\]
1414
1415<p>and define the group-relative advantage</p>
1416
1417\[\boxed{
1418
1419\hat A_i
1420
1421=
1422
1423\frac{R_i-\mu_q}{\sigma_q}.
1424
1425}\]
1426
1427<p>Under outcome supervision, the original GRPO formulation assigns the same response-level advantage to every token in the response:</p>
1428
1429\[\hat A_{i,t}=\hat A_i.\]
1430
1431<p>The clipped policy term is then the same as in PPO:</p>
1432
1433\[\min
1434
1435\left(
1436
1437\rho_{i,t}\hat A_i,
1438
1439\operatorname{clip}(\rho_{i,t},1-\epsilon,1+\epsilon)\hat A_i
1440
1441\right),\]
1442
1443<p>where</p>
1444
1445\[\rho_{i,t}
1446
1447=
1448
1449\frac{
1450
1451\pi_{\theta}(o_{i,t}\mid q,o_{i,\lt t})
1452
1453}{
1454
1455\pi_{\mathrm{old}}(o_{i,t}\mid q,o_{i,\lt t})
1456
1457}.\]
1458
1459<p>A common response-normalized form of the objective is</p>
1460
1461\[\begin{aligned}
1462
1463J_{\mathrm{GRPO}}(\theta)
1464
1465=
1466
1467\mathbb{E}
1468
1469\Bigg[
1470
1471\frac{1}{G}
1472
1473\sum_{i=1}^G
1474
1475\frac{1}{T_i}
1476
1477\sum_{t=1}^{T_i}
1478
1479\Bigg(&amp;
1480
1481\min
1482
1483\left(
1484
1485\rho_{i,t}\hat A_i,
1486
1487\operatorname{clip}(\rho_{i,t},1-\epsilon,1+\epsilon)\hat A_i
1488
1489\right)
1490
1491\\\\
1492
1493&amp;-\beta\hat D_{\mathrm{KL},i,t}
1494
1495\Bigg)
1496
1497\Bigg].
1498
1499\end{aligned}\]
1500
1501<p>The factor \(1/T_i\) gives each response equal total weight rather than giving longer responses more weight merely because they contain more tokens.</p>
1502
1503<h2 id="grpos-kl-term">GRPO’s KL Term</h2>
1504
1505<p>The original formulation uses the per-token estimator</p>
1506
1507\[\boxed{
1508
1509\hat D_{\mathrm{KL}}
1510
1511=
1512
1513\frac{\pi_{\mathrm{ref}}(a\mid s)}
1514
1515{\pi_{\theta}(a\mid s)}
1516
1517-
1518
1519\log
1520
1521\frac{\pi_{\mathrm{ref}}(a\mid s)}
1522
1523{\pi_{\theta}(a\mid s)}
1524
1525-1.
1526
1527}\]
1528
1529<p>Let</p>
1530
1531\[x=\frac{\pi_{\mathrm{ref}}(a\mid s)}
1532
1533{\pi_{\theta}(a\mid s)}.\]
1534
1535<p>Then</p>
1536
1537\[\hat D_{\mathrm{KL}}=x-\log x-1\geq 0,\]
1538
1539<p>
1539because \(\log x\leq x-1\) for \(x&gt;0\).</p>
1540
1541<p>If the action is sampled from the current policy, the expected value is the reverse KL divergence:</p>
1542
1543\[\begin{aligned}
1544
1545\mathbb{E}_{a\sim\pi_{\theta}}[\hat D_{\mathrm{KL}}]
1546
1547&amp;=
1548
1549\sum_a\pi_{\theta}(a)
1550
1551\left[
1552
1553\frac{\pi_{\mathrm{ref}}(a)}{\pi_{\theta}(a)}
1554
1555-
1556
1557\log\frac{\pi_{\mathrm{ref}}(a)}{\pi_{\theta}(a)}
1558
1559-1
1560
1561\right]
1562
1563\\\\
1564
1565&amp;=
1566
1567D_{\mathrm{KL}}\left(\pi_{\theta}\,\|\,\pi_{\mathrm{ref}}\right)
1568
1569\end{aligned}\]
1570
1571<p>For a fixed sampled token, define</p>
1572
1573\[u
1574
1575=
1576
1577\log\pi_{\theta}(a\mid s)
1578
1579-
1580
1581\log\pi_{\mathrm{ref}}(a\mid s).\]
1582
1583<p>Then \(x=e^{-u}\) and</p>
1584
1585\[\frac{\partial\hat D_{\mathrm{KL}}}{\partial u}
1586
1587=1-e^{-u}.\]
1588
1589<p>Therefore,</p>
1590
1591\[\boxed{
1592
1593\nabla_{\theta}\hat D_{\mathrm{KL}}
1594
1595=
1596
1597\left(
1598
15991-
1600
1601\frac{\pi_{\mathrm{ref}}(a\mid s)}
1602
1603{\pi_{\theta}(a\mid s)}
1604
1605\right)
1606
1607\nabla_{\theta}\log\pi_{\theta}(a\mid s).
1608
1609}\]
1610
1611<p>The policy-gradient portion of GRPO is otherwise identical to PPO. Its active gradient is</p>
1612
1613\[\boxed{
1614
1615\nabla_{\theta}J_{\mathrm{GRPO,policy}}
1616
1617=
1618
1619\mathbb{E}
1620
1621\left[
1622
1623\frac{1}{G}\sum_i\frac{1}{T_i}\sum_t
1624
1625m_{i,t}\rho_{i,t}\hat A_i
1626
1627\nabla_{\theta}
1628
1629\log\pi_{\theta}(o_{i,t}\mid q,o_{i,\lt t})
1630
1631\right].
1632
1633}\]
1634
1635<p>The algorithmic difference is where \(\hat A\) comes from:</p>
1636
1637\[\text{PPO: reward-to-go minus a learned value baseline,}\]
1638
1639\[\text{GRPO: response reward minus a same-prompt group baseline.}\]
1640
1641<h2 id="what-group-normalization-changes">What Group Normalization Changes</h2>
1642
1643<p>Group normalization has several consequences that are easy to miss.</p>
1644
1645<h3 id="the-advantages-are-centered">The advantages are centered</h3>
1646
1647<p>Ignoring the numerical epsilon,</p>
1648
1649\[\sum_{i=1}^G\hat A_i=0.\]
1650
1651<p>GRPO therefore performs a relative update within each prompt. Some responses are pushed up and others are pushed down.</p>
1652
1653<h3 id="positive-affine-reward-transformations-disappear">Positive affine reward transformations disappear</h3>
1654
1655<p>For \(R_i'=aR_i+b\) with \(a&gt;0\),</p>
1656
1657\[\frac{R_i'-\mu'}{\sigma'}
1658
1659=
1660
1661\frac{R_i-\mu}{\sigma}.\]
1662
1663<p>This makes GRPO insensitive to prompt-specific reward offsets and positive scales. It also means absolute reward magnitude is discarded after normalization.</p>
1664
1665<h3 id="equal-rewards-produce-no-reward-gradient">Equal rewards produce no reward gradient</h3>
1666
1667<p>If all responses receive the same reward, then every centered advantage is zero. The prompt supplies no reward-driven policy update. This can happen when all sampled answers are wrong, all are correct, or the reward function is too coarse.</p>
1668
1669<h3 id="with-two-samples-grpo-is-close-to-ranking">With two samples, GRPO is close to ranking</h3>
1670
1671<p>With \(G=2\), population standard deviation, and unequal rewards, the normalized advantages are \(+1\) and \(-1\). Only the ordering remains; the size of the reward gap disappears. A different standard-deviation convention changes the magnitude but not the direction.</p>
1672
1673<h3 id="the-ordinary-group-mean-contains-the-current-sample">The ordinary group mean contains the current sample</h3>
1674
1675<p>A textbook action-independent baseline should not depend on the sampled action whose score it multiplies. The group mean</p>
1676
1677\[\bar R=\frac{1}{G}\sum_jR_j\]
1678
1679<p>contains \(R_i\) itself. For the unnormalized centered estimator</p>
1680
1681\[\hat g
1682
1683=
1684
1685\frac{1}{G}
1686
1687\sum_{i=1}^G
1688
1689(R_i-\bar R)
1690
1691\nabla\log\pi(o_i),\]
1692
1693<p>independent sampling gives</p>
1694
1695\[\mathbb{E}[\hat g]
1696
1697=
1698
1699\frac{G-1}{G}
1700
1701\mathbb{E}
1702
1703\left[
1704
1705R\nabla\log\pi(o)
1706
1707\right].\]
1708
1709<p>Thus self-inclusion preserves the expected direction but scales the unnormalized estimator. A leave-one-out baseline</p>
1710
1711\[b_{-i}
1712
1713=
1714
1715\frac{1}{G-1}\sum_{j\neq i}R_j\]
1716
1717<p>is independent of response \(i\) conditional on the prompt and removes this particular factor. Dividing by the random group standard deviation introduces additional stochastic rescaling, so normalized GRPO is best understood as a practical relative estimator rather than a literal unbiased copy of REINFORCE.</p>
1718
1719<h2 id="a-small-numerical-grpo-example">A Small Numerical GRPO Example</h2>
1720
1721<p>Suppose four responses to the same prompt receive</p>
1722
1723\[R=[1,0,0,0].\]
1724
1725<p>The group mean is</p>
1726
1727\[\mu=0.25.\]
1728
1729<p>Using population standard deviation,</p>
1730
1731\[\sigma
1732
1733=
1734
1735\sqrt{
1736
1737\frac{(0.75)^2+3(-0.25)^2}{4}
1738
1739}
1740
1741\approx0.433.\]
1742
1743<p>The normalized advantages are</p>
1744
1745\[\hat A
1746
1747\approx
1748
1749[1.732,-0.577,-0.577,-0.577].\]
1750
1751<p>Every token in the successful response receives positive advantage, while tokens in the three unsuccessful responses receive negative advantage.</p>
1752
1753<p>Now let \(\epsilon=0.2\). For a token in the successful response with \(\rho=1.10\),</p>
1754
1755\[\rho\hat A
1756
1757=1.10\times1.732
1758
1759\approx1.905,\]
1760
1761<p>
1761and the gradient is active. If \(\rho=1.25\), PPO clipping limits the objective contribution to</p>
1762
1763\[1.2\times1.732
1764
1765\approx2.078,\]
1766
1767<p>and that sample supplies no further favorable policy gradient beyond the clipping boundary.</p>
1768
1769<h2 id="ppo-and-grpo-side-by-side">PPO and GRPO Side by Side</h2>
1770
1771<table>
1772  <thead>
1773    <tr>
1774      <th>Property</th>
1775      <th>PPO</th>
1776      <th>GRPO</th>
1777    </tr>
1778  </thead>
1779  <tbody>
1780    <tr>
1781      <td>Policy update</td>
1782      <td>Clipped probability-ratio objective</td>
1783      <td>Same clipped objective</td>
1784    </tr>
1785    <tr>
1786      <td>Baseline</td>
1787      <td>Learned value function</td>
1788      <td>Same-prompt group statistics</td>
1789    </tr>
1790    <tr>
1791      <td>Critic</td>
1792      <td>Normally required</td>
1793      <td>Not required</td>
1794    </tr>
1795    <tr>
1796      <td>Advantage granularity</td>
1797      <td>Usually token- or time-specific through GAE</td>
1798      <td>Often one outcome advantage per response</td>
1799    </tr>
1800    <tr>
1801      <td>Value loss</td>
1802      <td>Yes</td>
1803      <td>No</td>
1804    </tr>
1805    <tr>
1806      <td>Rollouts</td>
1807      <td>Ordinary trajectories</td>
1808      <td>Multiple responses for each prompt</td>
1809    </tr>
1810    <tr>
1811      <td>Main benefit</td>
1812      <td>Fine-grained learned credit assignment</td>
1813      <td>Lower memory and implementation complexity</td>
1814    </tr>
1815    <tr>
1816      <td>Main weakness</td>
1817      <td>Critic cost and value-estimation error</td>
1818      <td>Coarse credit assignment and zero-signal groups</td>
1819    </tr>
1820  </tbody>
1821</table>
1822
1823<p>A compact summary is</p>
1824
1825\[\boxed{
1826
1827\text{GRPO is a PPO-style update with a prompt-local Monte Carlo baseline.}
1828
1829}\]
1830
1831<p>It removes the critic. It does not remove the reward function, the rollout policy, PPO clipping, or necessarily the reference policy.</p>
1832
1833<h2 id="minimal-pytorch-style-implementation">Minimal PyTorch-Style Implementation</h2>
1834
1835<p>The following code contains the mathematical core. Production implementations additionally need distributed rollout collection, padding masks, mixed precision, optimizer state management, and careful metric logging.</p>
1836
1837<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
1838<span class="kn">from</span> <span class="nn">__future__</span> <span class="kn">import</span> <span class="n">annotations</span>
1839
1840<span class="kn">import</span> <span class="nn">torch</span>
1841
1842<span class="kn">from</span> <span class="nn">torch</span> <span class="kn">import</span> <span class="n">Tensor</span>
1843
1844
1845
1846<span class="k">def</span> <span class="nf">ppo_policy_loss</span><span class="p">(</span>
1847
1848    <span class="n">log_probs</span><span class="p">:</span> <span class="n">Tensor</span><span class="p">,</span>
1849
1850    <span class="n">old_log_probs</span><span class="p">:</span> <span class="n">Tensor</span><span class="p">,</span>
1851
1852    <span class="n">advantages</span><span class="p">:</span> <span class="n">Tensor</span><span class="p">,</span>
1853
1854    <span class="n">mask</span><span class="p">:</span> <span class="n">Tensor</span><span class="p">,</span>
1855
1856    \<span class="o">*</span><span class="p">,</span>
1857
1858    <span class="n">clip_epsilon</span><span class="p">:</span> <span class="nb">float</span> <span class="o">=</span> <span class="mf">0.2</span><span class="p">,</span>
1859
1860<span class="p">)</span> <span class="o">-&gt;</span> <span class="n">Tensor</span><span class="p">:</span>
1861
1862    <span class="s">"""Clipped PPO policy loss over valid tokens."""</span>
1863
1864    <span class="n">old_log_probs</span> <span class="o">=</span> <span class="n">old_log_probs</span><span class="p">.</span><span class="n">detach</span><span class="p">()</span>
1865
1866    <span class="n">advantages</span> <span class="o">=</span> <span class="n">advantages</span><span class="p">.</span><span class="n">detach</span><span class="p">()</span>
1867
1868    <span class="n">mask</span> <span class="o">=</span> <span class="n">mask</span><span class="p">.</span><span class="n">to</span><span class="p">(</span><span class="n">log_probs</span><span class="p">.</span><span class="n">dtype</span><span class="p">)</span>
1869
1870    <span class="n">ratio</span> <span class="o">=</span> <span class="n">torch</span><span class="p">.</span><span class="n">exp</span><span class="p">(</span><span class="n">log_probs</span> <span class="o">-</span> <span class="n">old_log_probs</span><span class="p">)</span>
1871
1872    <span class="n">unclipped</span> <span class="o">=</span> <span class="n">ratio</span> \<span class="o">*</span> <span class="n">advantages</span>
1873
1874    <span class="n">clipped</span> <span class="o">=</span> <span class="n">ratio</span><span class="p">.</span><span class="n">clamp</span><span class="p">(</span>
1875
1876        <span class="mf">1.0</span> <span class="o">-</span> <span class="n">clip_epsilon</span><span class="p">,</span>
1877
1878        <span class="mf">1.0</span> <span class="o">+</span> <span class="n">clip_epsilon</span><span class="p">,</span>
1879
1880    <span class="p">)</span> \<span class="o">*</span> <span class="n">advantages</span>
1881
1882    <span class="n">token_objective</span> <span class="o">=</span> <span class="n">torch</span><span class="p">.</span><span class="n">minimum</span><span class="p">(</span><span class="n">unclipped</span><span class="p">,</span> <span class="n">clipped</span><span class="p">)</span>
1883
1884    <span class="n">valid_tokens</span> <span class="o">=</span> <span class="n">mask</span><span class="p">.</span><span class="nb">sum</span><span class="p">().</span><span class="n">clamp_min</span><span class="p">(</span><span class="mf">1.0</span><span class="p">)</span>
1885
1886    <span class="k">return</span> <span class="o">-</span><span class="p">(</span><span class="n">token_objective</span> \<span class="o">*</span> <span class="n">mask</span><span class="p">).</span><span class="nb">sum</span><span class="p">()</span> <span class="o">/</span> <span class="n">valid_tokens</span>
1887
1888
1889
1890<span class="k">def</span> <span class="nf">grpo_loss</span><span class="p">(</span>
1891
1892    <span class="n">log_probs</span><span class="p">:</span> <span class="n">Tensor</span><span class="p">,</span>       <span class="c1"># [batch, group, time]
1893</span>
1894    <span class="n">old_log_probs</span><span class="p">:</span> <span class="n">Tensor</span><span class="p">,</span>   <span class="c1"># [batch, group, time]
1895</span>
1896    <span class="n">ref_log_probs</span><span class="p">:</span> <span class="n">Tensor</span><span class="p">,</span>   <span class="c1"># [batch, group, time]
1897</span>
1898    <span class="n">rewards</span><span class="p">:</span> <span class="n">Tensor</span><span class="p">,</span>         <span class="c1"># [batch, group]
1899</span>
1900    <span class="n">mask</span><span class="p">:</span> <span class="n">Tensor</span><span class="p">,</span>            <span class="c1"># [batch, group, time]
1901</span>
1902    \<span class="o">*</span><span class="p">,</span>
1903
1904    <span class="n">clip_epsilon</span><span class="p">:</span> <span class="nb">float</span> <span class="o">=</span> <span class="mf">0.2</span><span class="p">,</span>
1905
1906    <span class="n">kl_coefficient</span><span class="p">:</span> <span class="nb">float</span> <span class="o">=</span> <span class="mf">0.01</span><span class="p">,</span>
1907
1908    <span class="n">std_epsilon</span><span class="p">:</span> <span class="nb">float</span> <span class="o">=</span> <span class="mf">1e-6</span><span class="p">,</span>
1909
1910<span class="p">)</span> <span class="o">-&gt;</span> <span class="n">Tensor</span><span class="p">:</span>
1911
1912    <span class="s">"""Outcome-supervised, response-normalized GRPO loss."""</span>
1913
1914    <span class="n">old_log_probs</span> <span class="o">=</span> <span class="n">old_log_probs</span><span class="p">.</span><span class="n">detach</span><span class="p">()</span>
1915
1916    <span class="n">ref_log_probs</span> <span class="o">=</span> <span class="n">ref_log_probs</span><span class="p">.</span><span class="n">detach</span><span class="p">()</span>
1917
1918    <span class="n">rewards</span> <span class="o">=</span> <span class="n">rewards</span><span class="p">.</span><span class="n">detach</span><span class="p">()</span>
1919
1920    <span class="n">mask</span> <span class="o">=</span> <span class="n">mask</span><span class="p">.</span><span class="n">to</span><span class="p">(</span><span class="n">log_probs</span><span class="p">.</span><span class="n">dtype</span><span class="p">)</span>
1921
1922    <span class="n">group_mean</span> <span class="o">=</span> <span class="n">rewards</span><span class="p">.</span><span class="n">mean</span><span class="p">(</span><span class="n">dim</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">keepdim</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
1923
1924    <span class="n">group_variance</span> <span class="o">=</span> <span class="p">(</span><span class="n">rewards</span> <span class="o">-</span> <span class="n">group_mean</span><span class="p">).</span><span class="n">square</span><span class="p">().</span><span class="n">mean</span><span class="p">(</span>
1925
1926        <span class="n">dim</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span>
1927
1928        <span class="n">keepdim</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span>
1929
1930    <span class="p">)</span>
1931
1932    <span class="n">group_std</span> <span class="o">=</span> <span class="n">torch</span><span class="p">.</span><span class="n">sqrt</span><span class="p">(</span><span class="n">group_variance</span> <span class="o">+</span> <span class="n">std_epsilon</span><span class="p">)</span>
1933
1934    <span class="n">response_advantage</span> <span class="o">=</span> <span class="p">(</span>
1935
1936        <span class="p">(</span><span class="n">rewards</span> <span class="o">-</span> <span class="n">group_mean</span><span class="p">)</span> <span class="o">/</span> <span class="n">group_std</span>
1937
1938    <span class="p">).</span><span class="n">detach</span><span class="p">()</span>
1939
1940    <span class="n">token_advantage</span> <span class="o">=</span> <span class="n">response_advantage</span><span class="p">.</span><span class="n">unsqueeze</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">)</span>
1941
1942    <span class="n">ratio</span> <span class="o">=</span> <span class="n">torch</span><span class="p">.</span><span class="n">exp</span><span class="p">(</span><span class="n">log_probs</span> <span class="o">-</span> <span class="n">old_log_probs</span><span class="p">)</span>
1943
1944    <span class="n">unclipped</span> <span class="o">=</span> <span class="n">ratio</span> \<span class="o">*</span> <span class="n">token_advantage</span>
1945
1946    <span class="n">clipped</span> <span class="o">=</span> <span class="n">ratio</span><span class="p">.</span><span class="n">clamp</span><span class="p">(</span>
1947
1948        <span class="mf">1.0</span> <span class="o">-</span> <span class="n">clip_epsilon</span><span class="p">,</span>
1949
1950        <span class="mf">1.0</span> <span class="o">+</span> <span class="n">clip_epsilon</span><span class="p">,</span>
1951
1952    <span class="p">)</span> \<span class="o">*</span> <span class="n">token_advantage</span>
1953
1954    <span class="n">policy_token_loss</span> <span class="o">=</span> <span class="o">-</span><span class="n">torch</span><span class="p">.</span><span class="n">minimum</span><span class="p">(</span><span class="n">unclipped</span><span class="p">,</span> <span class="n">clipped</span><span class="p">)</span>
1955
1956    <span class="n">log_ref_over_policy</span> <span class="o">=</span> <span class="n">ref_log_probs</span> <span class="o">-</span> <span class="n">log_probs</span>
1957
1958    <span class="n">kl_token_loss</span> <span class="o">=</span> <span class="p">(</span>
1959
1960        <span class="n">torch</span><span class="p">.</span><span class="n">exp</span><span class="p">(</span><span class="n">log_ref_over_policy</span><span class="p">)</span>
1961
1962        <span class="o">-</span> <span class="n">log_ref_over_policy</span>
1963
1964        <span class="o">-</span> <span class="mf">1.0</span>
1965
1966    <span class="p">)</span>
1967
1968    <span class="n">token_loss</span> <span class="o">=</span> <span class="n">policy_token_loss</span> <span class="o">+</span> <span class="n">kl_coefficient</span> \<span class="o">*</span> <span class="n">kl_token_loss</span>
1969
1970    <span class="n">token_count</span> <span class="o">=</span> <span class="n">mask</span><span class="p">.</span><span class="nb">sum</span><span class="p">(</span><span class="n">dim</span><span class="o">=-</span><span class="mi">1</span><span class="p">).</span><span class="n">clamp_min</span><span class="p">(</span><span class="mf">1.0</span><span class="p">)</span>
1971
1972    <span class="n">response_loss</span> <span class="o">=</span> <span class="p">(</span><span class="n">token_loss</span> \<span class="o">*</span> <span class="n">mask</span><span class="p">).</span><span class="nb">sum</span><span class="p">(</span><span class="n">dim</span><span class="o">=-</span><span class="mi">1</span><span class="p">)</span> <span class="o">/</span> <span class="n">token_count</span>
1973
1974    <span class="k">return</span> <span class="n">response_loss</span><span class="p">.</span><span class="n">mean</span><span class="p">()</span>
1975
1976</code></pre></div></div>
1977
1978<p>The essential implementation rules are visible in the code:</p>
1979
1980<ul>
1981  <li>
1982    <p>Compute probability ratios as exponentiated log-probability differences.</p>
1983  </li>
1984  <li>
1985    <p>Detach old log-probabilities and advantages.</p>
1986  </li>
1987  <li>
1988    <p>Normalize GRPO rewards independently within each prompt group.</p>
1989  </li>
1990  <li>
1991    <p>Apply the response mask before averaging.</p>
1992  </li>
1993  <li>
1994    <p>Decide deliberately whether to weight responses equally or tokens equally.</p>
1995  </li>
1996</ul>
1997
1998<h2 id="common-failure-modes">Common Failure Modes</h2>
1999
2000<h3 id="confusing-objective-and-loss-signs">Confusing objective and loss signs</h3>
2001
2002<p>Papers usually write an objective to maximize. Deep-learning libraries usually minimize. The implementation therefore needs a leading minus sign on the clipped policy objective.</p>
2003
2004<h3 id="dividing-log-probabilities">Dividing log-probabilities</h3>
2005
2006<p>The correct ratio is</p>
2007
2008\[\rho=\exp(\log\pi_{\theta}-\log\pi_{\mathrm{old}}),\]
2009
2010<p>not \(\log\pi_{\theta}/\log\pi_{\mathrm{old}}\).</p>
2011
2012<h3 id="backpropagating-through-rollout-quantities">Backpropagating through rollout quantities</h3>
2013
2014<p>Old log-probabilities, sampled rewards, returns, and advantages are fixed data during the actor update. They should not retain a gradient path to trainable rollout models.</p>
2015
2016<h3 id="confusing-the-old-and-reference-policies">Confusing the old and reference policies</h3>
2017
2018<p>The old policy is the denominator of the PPO ratio. The reference policy is the target of KL regularization. Replacing one role with the other changes the algorithm.</p>
2019
2020<h3 id="refreshing-the-old-policy-inside-the-optimization-epoch">Refreshing the old policy inside the optimization epoch</h3>
2021
2022<p>The old policy should remain fixed while the rollout batch is reused for several minibatch updates. Otherwise the denominator changes and the clipping interpretation no longer matches the collected data.</p>
2023
2024<h3 id="treating-clipping-as-a-global-kl-guarantee">Treating clipping as a global KL guarantee</h3>
2025
2026<p>PPO clipping limits the incentive contributed by individual sampled actions. It does not mathematically guarantee that the complete policy remains inside a fixed KL ball. Learning rate, number of epochs, and correlated parameter changes still matter.</p>
2027
2028<h3 id="normalizing-grpo-rewards-across-unrelated-prompts">Normalizing GRPO rewards across unrelated prompts</h3>
2029
2030<p>The defining comparison is within the group generated for one prompt. Batch-wide normalization mixes prompt difficulty and changes the meaning of the advantage.</p>
2031
2032<h3 id="ignoring-zero-variance-groups">Ignoring zero-variance groups</h3>
2033
2034<p>When every response in a group receives the same reward, the normalized advantages vanish. Monitoring the fraction of zero-variance groups is therefore an important GRPO diagnostic.</p>
2035
2036<h2 id="a-unifying-view-of-post-training">A Unifying View of Post-Training</h2>
2037
2038<p>Many post-training methods can be understood as weighted log-likelihood updates:</p>
2039
2040\[\nabla_{\theta}J
2041
2042=
2043
2044\mathbb{E}
2045
2046\left[
2047
2048 c(q,o,t)
2049
2050\nabla_{\theta}
2051
2052\log\pi_{\theta}(o_t\mid q,o_{&lt;t})
2053
2054\right].\]
2055
2056<p>The coefficient \(c\) defines the method.</p>
2057
2058<p>For supervised fine-tuning,</p>
2059
2060\[c=1\]
2061
2062<p>on selected demonstration tokens.</p>
2063
2064<p>For REINFORCE,</p>
2065
2066\[c=G_t.\]
2067
2068<p>For advantage actor-critic methods,</p>
2069
2070\[c=\hat A_t.\]
2071
2072<p>For active PPO samples,</p>
2073
2074\[c=m_t\rho_t\hat A_t.\]
2075
2076<p>For outcome-supervised GRPO,</p>
2077
2078\[c=m_{i,t}\rho_{i,t}
2079
2080\frac{R_i-\mu_q}{\sigma_q}.\]
2081
2082<p>This perspective makes the algorithms less mysterious. The neural-network backpropagation is ordinary log-likelihood backpropagation. Most of the conceptual work lies in constructing a useful, low-variance, stable coefficient for every sampled token.</p>
2083
2084<h2 id="conclusion">Conclusion</h2>
2085
2086<p>The path from policy gradients to PPO and GRPO consists of a small number of reusable ideas.</p>
2087
2088<p>First, the log-derivative trick turns the gradient of expected reward into reward-weighted log-probability gradients. Second, causality removes past rewards: for \(k\lt t\),</p>
2089
2090\[\mathbb{E}
2091
2092\left[
2093
2094 r_k\nabla\log\pi(a_t\mid s_t)
2095
2096\right]=0,\]
2097
2098<p>because the expected score of the newly sampled action is zero after conditioning on the history. This produces reward-to-go.</p>
2099
2100<p>Third, the same score identity permits action-independent baselines, leading to advantages. PPO estimates those advantages with a critic, often through GAE, and stabilizes data reuse with an old-policy ratio and clipping. GRPO keeps the clipped PPO machinery but replaces the learned value baseline with a relative baseline constructed from several responses to the same prompt.</p>
2101
2102<p>At the start of either update, when the current and old policies coincide, the central instruction is simple:</p>
2103
2104\[\boxed{
2105
2106\text{Increase the probability of positive-advantage tokens and decrease the probability of negative-advantage tokens.}
2107
2108}\]
2109
2110<p>PPO and GRPO differ mainly in how they decide which sampled tokens deserve those positive or negative weights, and in how they prevent repeated o
2110ptimization from moving the policy too aggressively.</p>
2111
2112<h2 id="references">References</h2>
2113
2114<p><span class="reference">[Wil92]</span> R. J. Williams. [Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning](https://link.springer.com/article/10.1007/BF00992696). <em>*Machine Learning*</em>, 1992.</p>
2115
2116<p><span class="reference">[Sut99]</span> R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour. [Policy Gradient Methods for Reinforcement Learning with Function Approximation](https://proceedings.neurips.cc/paper/1999/hash/464d828b85b0bed98e80ade0a5c43b0f-Abstract.html). <em>*NeurIPS*</em>, 1999.</p>
2117
2118<p><span class="reference">[Sch15a]</span> J. Schulman, S. Levine, P. Moritz, M. I. Jordan, and P. Abbeel. [Trust Region Policy Optimization](https://arxiv.org/abs/1502.05477). 2015.</p>
2119
2120<p><span class="reference">[Sch15b]</span> J. Schulman, P. Moritz, S. Levine, M. I. Jordan, and P. Abbeel. [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://arxiv.org/abs/1506.02438). 2015.</p>
2121
2122<p><span class="reference">[Sch17]</span> J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. [Proximal Policy Optimization Algorithms](https://arxiv.org/abs/1707.06347). 2017.</p>
2123
2124<p><span class="reference">[Sha24]</span> Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo. [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://arxiv.org/abs/2402.03300). 2024.</p>
2125
2126  </div><div id="disqus_thread"></div>
2127  
2127<script>
2128    var disqus_config = function () {
2129      this.page.url = 'https://www.norange.ai/post-training/2026/08/24/post-training-primer.html';
2130      this.page.identifier = 'https://www.norange.ai/post-training/2026/08/24/post-training-primer.html';
2131    };
2132
2133    (function() {
2134      var d = document, s = d.createElement('script');
2135
2136      s.src = 'https://nikitadurasov-github-io.disqus.com/embed.js';
2137
2138      s.setAttribute('data-timestamp', +new Date());
2139      (d.head || d.body).appendChild(s);
2140    })();
2141  </script>
2141
2142  <noscript>Please enable JavaScript to view the <a href="https://disqus.com/?ref_noscript" rel="nofollow">comments powered by Disqus.</a></noscript><a class="u-url" href="/post-training/2026/08/24/post-training-primer.html" hidden></a>
2143</article>
2144
2145<script>
2146(function () {
2147  var content = document.querySelector(".post-content");
2148  var nav = document.querySelector(".post-toc");
2149  if (!content) return;
2150
2151  var headings = Array.prototype.slice
2152    .call(content.querySelectorAll("h2, h3, h4, h5, h6"))
2153    .filter(function (heading) {
2154      return heading.textContent.trim().length > 0;
2155    });
2156
2157  if (!headings.length) return;
2158
2159  function slugify(text) {
2160    return text
2161      .toLowerCase()
2162      .replace(/[^\w\s-]/g, "")
2163      .trim()
2164      .replace(/\s+/g, "-");
2165  }
2166
2167  var usedIds = {};
2168
2169  // Labels are captured before anchors are appended, so the "##" markers below
2170  // never leak into the table of contents.
2171  var sections = headings.map(function (heading) {
2172    var label = heading.textContent.trim();
2173
2174    if (!heading.id) {
2175      var base = slugify(label) || "section";
2176      var id = base;
2177      var suffix = 1;
2178      while (document.getElementById(id) || usedIds[id]) {
2179        id = base + "-" + suffix++;
2180      }
2181      heading.id = id;
2182    }
2183    usedIds[heading.id] = true;
2184
2185    return {
2186      heading: heading,
2187      level: parseInt(heading.tagName.charAt(1), 10),
2188      label: label
2189    };
2190  });
2191
2192  function copyText(text) {
2193    if (navigator.clipboard && navigator.clipboard.writeText) {
2194      navigator.clipboard.writeText(text);
2195      return;
2196    }
2197    var field = document.createElement("textarea");
2198    field.value = text;
2199    field.setAttribute("readonly", "");
2200    field.style.position = "fixed";
2201    field.style.opacity = "0";
2202    document.body.appendChild(field);
2203    field.select();
2204    try {
2205      document.execCommand("copy");
2206    } catch (err) {}
2207    document.body.removeChild(field);
2208  }
2209
2210  var copiedTimer;
2211
2212  sections.forEach(function (section) {
2213    var heading = section.heading;
2214
2215    var anchor = document.createElement("a");
2216    anchor.className = "heading-anchor";
2217    anchor.href = "#" + heading.id;
2218    anchor.setAttribute("aria-label", "Copy link to this section");
2219    anchor.textContent = new Array(section.level + 1).join("+");
2220
2221    heading.firstChild.nodeValue = " " + heading.firstChild.nodeValue;
2222    heading.insertBefore(anchor, heading.firstChild);
2223    heading.classList.add("is-linkable");
2224
2225    heading.addEventListener("click", function (event) {
2226      if (String(window.getSelection())) return;
2227      event.preventDefault();
2228
2229      copyText(
2230        location.origin + location.pathname + "#" + heading.id
2231      );
2232      history.replaceState(null, "", "#" + heading.id);
2233
2234      window.clearTimeout(copiedTimer);
2235      Array.prototype.forEach.call(
2236        content.querySelectorAll(".heading-anchor.is-copied"),
2237        function (node) {
2238          node.classList.remove("is-copied");
2239        }
2240      );
2241      anchor.classList.add("is-copied");
2242      copiedTimer = window.setTimeout(function () {
2243        anchor.classList.remove("is-copied");
2244      }, 1600);
2245    });
2246  });
2247
2248  if (!nav || sections.length < 3) return;
2249
2250  var root = document.createElement("ul");
2251
2252  var stack = [{ level: 0, list: root }];
2253  var links = [];
2254
2255  sections.forEach(function (section) {
2256    var heading = section.heading;
2257    var level = section.level;
2258
2259    while (stack.length > 1 && level <= stack[stack.length - 1].level) {
2260      stack.pop();
2261    }
2262
2263    var item = document.createElement("li");
2264
2265    var link = document.createElement("a");
2266    link.href = "#" + heading.id;
2267    link.textContent = section.label;
2268    item.appendChild(link);
2269
2270    stack[stack.length - 1].list.appendChild(item);
2271
2272    var sublist = document.createElement("ul");
2273    item.appendChild(sublist);
2274    stack.push({ level: level, list: sublist });
2275
2276    links.push({ heading: heading, link: link, item: item });
2277  });
2278
2279  Array.prototype.forEach.call(root.querySelectorAll("ul"), function (list) {
2280    if (!list.children.length) list.parentNode.removeChild(list);
2281  });
2282
2283  nav.appendChild(root);
2284  nav.classList.add("post-toc--ready");
2285
2286  var active = null;
2287
2288  function keepVisible(link) {
2289    var navBox = nav.getBoundingClientRect();
2290    var linkBox = link.getBoundingClientRect();
2291    if (linkBox.top < navBox.top) {
2292      nav.scrollTop -= navBox.top - linkBox.top + 24;
2293    } else if (linkBox.bottom > navBox.bottom) {
2294      nav.scrollTop += linkBox.bottom - navBox.bottom + 24;
2295    }
2296  }
2297
2298  function highlight() {
2299    var current = links[0];
2300    for (var i = 0; i < links.length; i++) {
2301      if (links[i].heading.getBoundingClientRect().top > 120) break;
2302      current = links[i];
2303    }
2304
2305    if (current === active) return;
2306    active = current;
2307
2308    Array.prototype.forEach.call(
2309      nav.querySelectorAll(".is-active, .is-parent"),
2310      function (node) {
2311        node.classList.remove("is-active", "is-parent");
2312      }
2313    );
2314
2315    current.link.classList.add("is-active");
2316
2317    var parent = current.item.parentNode;
2318    while (parent && parent !== root) {
2319      if (parent.tagName === "LI") {
2320        var parentLink = parent.querySelector("a");
2321        if (parentLink) parentLink.classList.add("is-parent");
2322      }
2323      parent = parent.parentNode;
2324    }
2325
2326    keepVisible(current.link);
2327  }
2328
2329  var ticking = false;
2330  function onScroll() {
2331    if (ticking) return;
2332    ticking = true;
2333    window.requestAnimationFrame(function () {
2334      ticking = false;
2335      highlight();
2336    });
2337  }
2338
2339  window.addEventListener("scroll", onScroll, { passive: true });
2340  window.addEventListener("resize", onScroll);
2341  highlight();
2342})();
2343</script>
2343
2344
2345<script>
2346fetch("/assets/views.json")
2347  .then((res) => res.json())
2348  .then((data) => {
2349    const path = window.location.pathname.replace(/\/$/, "");
2350    const basename = path.split("/").pop(); // ex: "uncertainty-estimation-intro.html", sums up all views ending with this basename
2351
2352    let total = 0;
2353    for (const [k, v] of Object.entries(data)) {
2354      if (typeof v === "number" && k.endsWith("/" + basename)) {
2355        total += v;
2356      }
2357    }
2358
2359  document.getElementById("ga_site_pv").textContent = total.toLocaleString();
2360    })
2361    .catch((err) => {
2362      console.error("Failed to load total view count", err);
2363      document.getElementById("ga_site_pv").textContent = "N/A";
2364    });
2365  </script>
2365
2366        </div>
2367      </div>
2368    </main><footer class="site-footer h-card">
2369  <data class="u-url" href="/"></data>
2370
2371  <div class="wrapper">
2372
2373    <div class="footer-col-wrapper">
2374      <div class="footer-col">
2375        <!-- <p class="feed-subscribe">
2376          <a href="/feed.xml">
2377            <svg class="svg-icon orange">
2378              <use xlink:href="/assets/minima-social-icons.svg#rss"></use>
2379            </svg><span>Subscribe</span>
2380          </a>
2381        </p> -->
2382      </div>
2383      <div class="footer-col">
2384        <p style="opacity: 0.2">Thoughts on deep learning, tech, and whatever else seems interesting.</p>
2385      </div>
2386    </div>
2387
2388    <div class="social-links"><style>
2389  .google-scholar-icon {
2390    transform: rotate(180deg);
2391    transform-origin: center;
2392    display: inline-block;
2393  }
2394</style>
2395
2396<ul class="social-media-list">
2397  <!--<li><a rel="me" href="https://github.com/nikitadurasov" class="svg-a" title="nikitadurasov"><svg class="svg-icon blue"><use xlink:href="/assets/minima-social-icons.svg#github"></use></svg></a></li><li><a rel="me" href="https://www.linkedin.com/in/durasovnikita" class="svg-a" title="durasovnikita"><svg class="svg-icon blue"><use xlink:href="/assets/minima-social-icons.svg#linkedin"></use></svg></a></li><li><a rel="me" href="https://twitter.com/norange_ai" class="svg-a" title="norange_ai"><svg class="svg-icon blue"><use xlink:href="/assets/minima-social-icons.svg#twitter"></use></svg></a></li>--><li><a rel="me" href="https://github.com/nikitadurasov" class="svg-a" title="nikitadurasov"><svg class="svg-icon blue"><use xlink:href="/assets/minima-social-icons.svg#github"></use></svg></a></li><li><a rel="me" href="https://www.linkedin.com/in/durasovnikita" class="svg-a" title="durasovnikita"><svg class="svg-icon blue"><use xlink:href="/assets/minima-social-icons.svg#linkedin"></use></svg></a></li><li><a rel="me" href="https://twitter.com/norange_ai" class="svg-a" title="norange_ai"><svg class="svg-icon blue"><use xlink:href="/assets/minima-social-icons.svg#twitter"></use></svg></a></li><!----><li>
2398    <a rel="me" href="https://scholar.google.com/citations?user=KMMvTfcAAAAJ%26hl" class="svg-a" title="Google Scholar Profile for KMMvTfcAAAAJ&amp;hl">
2399      <svg class="svg-icon blue google-scholar-icon">
2400        <use xlink:href="/assets/minima-social-icons.svg#google-scholar"></use>
2401      </svg>
2402    </a>
2403  </li></ul>
2404
2405
2406</div>
2407
2408  </div>
2409
2410</footer>
2411<!-- jQuery first, then Popper.js, then Bootstrap JS. -->
2412    
2412<script src="https://code.jquery.com/jquery-3.3.1.slim.min.js" integrity="sha384-q8i/X+965DzO0rT7abK41JStQIAqVgRVzpbzo5smXKp4YfRvH+8abtTE1Pi6jizo" crossorigin="anonymous"></script>
2412
2413    
2413<script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.14.7/umd/popper.min.js" integrity="sha384-UO2eT0CpHqdSJQ6hJty5KVphtPhzWj9WO1clHTMGa3JDZwrnQq4sF86dIHNDz0W1" crossorigin="anonymous"></script>
2413
2414    
2414<script src="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/js/bootstrap.min.js" integrity="sha384-JjSmVgyd0p3pXB1rRibZUAYoIIy6OrQ6VrjIEaFf/nJGzIxFDsf4x0xIM+B07jRM" crossorigin="anonymous"></script>
2414
2415
2416  </body>
2417
2418</html>

Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.