https://timothymin.github.io/RL/value-function/

1<!DOCTYPE HTML>
2<html lang="en">
3
4<head>
5  <!-- Google tag (gtag.js) -->

5
6  <script async src="https://www.googletagmanager.com/gtag/js?id=

6G-E00L9PT3V9

6"></script>
7  

7<script>
8    

8window.dataLayer = window.dataLayer || [];
9    function gtag(){dataLayer.push(arguments);}
10    gtag('js', new Date());
11
12    gtag('config', '

12G-E00L9PT3V9

12');
13  

13</script>

13
14
15  <!-- mathjax support -->
16  

16<script src="https://polyfill.io/v3/polyfill.min.js?features=es6"></script>

16
17    

17<script>
18    window.MathJax = {
19      tex: {
20        inlineMath: [['$', '$']]
21        // displayMath: [['$$', '$$']]
22      }
23    };
24  </script>

24
25  

25<script id="MathJax-script" async
26    src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js">
27  </script>

27
28  
29  <title>Joonkyu Min</title>
30
31  <meta content="text/html; charset=utf-8" http-equiv="Content-Type">
32
33  <meta name="author" content="Joonkyu Min" />
34  <meta name="viewport" content="width=device-width, initial-scale=1">
35
36  <link rel="stylesheet" type="text/css" href="/style.css" />
37  <link rel="canonical" href="https://timothymin.github/RL/value-function/">
38  <link href="https://fonts.googleapis.com/css?family=Lato:400,700,400italic,700italic" rel="stylesheet" type="text/css">
39
40  
41</head>
42
43<body>
44  <table style="width:100%;max-width:800px;border:0px;border-spacing:0px;border-collapse:separate;margin-right:auto;margin-left:auto;">
45    <tr style="padding:0px">
46      <td style="padding:0px">
47        <table style="width:100%;border:0px;border-spacing:0px;border-collapse:separate;margin-right:auto;margin-left:auto;">
48          <tr style="padding:0px">
49            <td style="padding:2.5%;width:60%;vertical-align:middle">
50              <h1>
51                Joonkyu Min
52              </h1>
53              <p>
54                I am an undergraduate student at Seoul National University, majoring in Electrical and Computer Engineering. 
55                I am currently an intern at <a href="https://clvrai.com">CLVR Lab</a>, where I work on robot safety under supervision of <a href="https://clvrai.com/web_lim/">Joseph J. Lim</a>.
56              </p>
57              <p style="text-align:center">
58                <a target="_blank" href="https://mailhide.io/e/B8TyS"> Email</a> &nbsp;/&nbsp;
59                <a href="https://github.com/timothymin">GitHub</a> &nbsp;/&nbsp;
60                <a href="https://www.linkedin.com/in/joonkyumin"> LinkedIn </a> &nbsp;/&nbsp;
61              </p>
62            </td>
63            <!-- <td style="padding:2.5%;width:40%;max-width:40%">
64              <img style="width:100%;max-width:100%" alt="profile photo" src="images/profile_pic.jpg">
65            </td> -->
66          </tr>
67        </table>
68
69        <table style="width:100%;border:0px;border-spacing:0px;border-collapse:separate;margin-right:auto;margin-left:auto;">
70          <tr>
71            <td style="padding:2.5%;width:100%;vertical-align:middle">
72              <article>
73                <hr>
74                <h1>What is a value function?</h1>
75                <p style="font-size:small; color:gray;">April 11, 2025</p>
76                <p>In RL, we try to maximize the long-term collected discounted rewards, or return.</p>
77
78\[G^\pi_t = \sum^\infty_{i=0}\gamma^i r_{t+i+1} \le \frac{\sup r}{1-\gamma}\]
79
80<p><strong>Value</strong> in MDP is the expected return of a particular state, following a given policy.</p>
81
82<p>There are typically two value functions, state value function $V$, and state-action value function $Q$ .</p>
83
84<ul>
85  <li><strong>State value function</strong></li>
86</ul>
87
88<p>State value function is a mapping from state $s$ to the expected return starting from $s$, following a particular policy $\pi$ .</p>
89
90\[V^\pi(s)=\mathbb{E}^\pi[G_t|s_{t}=s]\]
91
92<ul>
93  <li><strong>State action value function</strong></li>
94</ul>
95
96<p>State-action value function, or <strong>Q-function</strong>, is a mapping from state $s$ and action $a$ to the expected return starting from state $s$ and action $a$, following a particular policy $\pi$ .</p>
97
98\[Q^\pi(s)=\mathbb{E}^\pi[G_t|s_{t}=s,a_{t}=a]\]
99
100<p>Due to the Markovian property, value function holds the following 1-step transition property.</p>
101
102\[\begin{align}
103V^\pi(s)&amp;=\mathbb{E}_{a\sim \pi(\cdot|s), s'\sim p(\cdot|s,a)}[r+\gamma V^\pi(s')|s_{0}=s] \\
104Q^\pi(s)&amp;=\mathbb{E}_{a'\sim \pi(\cdot|s'), s'\sim p(\cdot|s,a)}[r+\gamma Q^\pi(s',a')|s_{0}=s,a_{0}=a]
105\end{align}\]
106
107<p>This leads to the famous <strong>Bellman equation</strong>.</p>
108
109<hr />
110
111<p><strong>Reference</strong></p>
112
113<p>R. Sutton, A. Barto, Reinforcement Learning: An Introduction. The MIT Press, 2018.</p>
114
115              </article>
116            </td>
117          </tr>
118        </table>
119        
120        <table style="width:100%;border:0px;border-spacing:0px;border-collapse:separate;margin-right:auto;margin-left:auto;">
121          <tr>
122            <td style="padding:0px">
123              <br>
124              <p style="text-align:center;font-size:small;">

125                Design and source code from <a style="font-size:small;" href="https://jonbarron.info">Jon Barron's website</a>
126              </p>
127            </td>
128          </tr>
129        </table>
130      </td>
131    </tr>
132  </table>
133</body>
134
135</html>