1<!DOCTYPE html> 2<html lang="en"> 3 4 <head> 5 6 7 8 9 10 11 12 13 14 15 <meta charset="utf-8"> 16 <meta http-equiv="X-UA-Compatible" content="IE=edge"> 17 <meta name="viewport" content="width=device-width, initial-scale=1"> 18 19 <title>Soft Actor Critic ë¶ì</title> 20 21 <meta name="description" content="Off Policy Trainì íµí´, Sample Efficiency를 ëì¬ë³´ì."> 22 23 <meta content="Heecheol Cho's Deep Learning Blog" property="og:site_name"> 24 25 <meta content="Soft Actor Critic ë¶ì" property="og:title"> 26 27 28 <meta content="article" property="og:type"> 29 30 31 <meta content="Off Policy Trainì íµí´, Sample Efficiency를 ëì¬ë³´ì." property="og:description"> 32 33 34 <meta content="https://hccho2.github.io//2020/04/19/SAC-analysis.html" property="og:url"> 35 36 37 <meta content="2020-04-19T06:53:34+00:00" property="article:published_time"> 38 <meta content="https://hccho2.github.io//about/" property="article:author"> 39 40 41 42 43 44 45 <link rel="shortcut icon" href="/assets/Cube.ico"> 46 <link rel="stylesheet" href="/assets/css/main.css"> 47 <link rel="canonical" href="https://hccho2.github.io//2020/04/19/SAC-analysis.html"> 48 49 <!-- For Latex --> 50
50<script src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML" type="text/javascript"></script>
50 51 52 <!-- Google Analytics --> 53
53<script>
vendor: 375 bytes, lines 53-59
53 54 (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ 55 (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), 56 m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) 57 })(window,document,'script','https://www.google-analytics.com/analytics.js','ga'); 58 59 ga('create', '
59UA-8161570-6
vendor: 48 bytes, lines 59-61
59', 'auto'); 60 ga('send', 'pageview'); 61
61</script>
61 62 63 <!-- For Facebook share button --> 64 <div id="fb-root"></div> 65
65<script> 66 (function(d, s, id) { 67 var js, fjs = d.getElementsByTagName(s)[0]; 68 if (d.getElementById(id)) return; 69 js = d.createElement(s); js.id = id; 70 js.src = "//connect.facebook.net/en_US/sdk.js#xfbml=1&version=v2.9"; 71 fjs.parentNode.insertBefore(js, fjs); 72 }(document, 'script', 'facebook-jssdk')); 73 </script>
73 74 75 <!-- Twitter cards --> 76 <meta name="twitter:site" content="@"> 77 <meta name="twitter:creator" content="@"> 78 <meta name="twitter:title" content="Soft Actor Critic ë¶ì"> 79 80 81 <meta name="twitter:description" content="<blockquote> 82 <p>Off Policy Trainì íµí´, Sample Efficiency를 ëì¬ë³´ì.</p> 83</blockquote> 84 85"> 86 87 88 89 <meta name="twitter:card" content="summary"> 90 <meta name="twitter:image" content=""> 91 92 <!-- end of Twitter cards --> 93 94</head> 95 96 97 <body> 98 99 <header class="site-header"> 100 101 <div class="wrapper"><a class="site-title" rel="author" href="/">Heecheol Cho's Deep Learning Blog</a><nav class="site-nav"> 102 <input type="checkbox" id="nav-trigger" class="nav-trigger" /> 103 <label for="nav-trigger"> 104 <span class="menu-icon"> 105 <svg viewBox="0 0 18 15" width="18px" height="15px"> 106 <path d="M18,1.484c0,0.82-0.665,1.484-1.484,1.484H1.484C0.665,2.969,0,2.304,0,1.484l0,0C0,0.665,0.665,0,1.484,0 h15.032C17.335,0,18,0.665,18,1.484L18,1.484z M18,7.516C18,8.335,17.335,9,16.516,9H1.484C0.665,9,0,8.335,0,7.516l0,0 c0-0.82,0.665-1.484,1.484-1.484h15.032C17.335,6.031,18,6.696,18,7.516L18,7.516z M18,13.516C18,14.335,17.335,15,16.516,15H1.484 C0.665,15,0,14.335,0,13.516l0,0c0-0.82,0.665-1.483,1.484-1.483h15.032C17.335,12.031,18,12.695,18,13.516L18,13.516z"/> 107 </svg> 108 </span> 109 </label> 110 111 <div class="trigger"><a class="page-link" href="/about/">About</a></div> 112 </nav></div> 113</header> 114 115 <main class="page-content" aria-label="Content"> 116 <div class="wrapper"> 117 <article class="post" itemscope itemtype="http://schema.org/BlogPosting"> 118 119 <header class="post-header"> 120 <h1 class="post-title" itemprop="name headline">Soft Actor Critic ë¶ì</h1> 121 <p class="post-meta"> 122 123 <time datetime="2020-04-19T06:53:34+00:00" itemprop="datePublished"> 124 125 Apr 19, 2020 126 </time> 127 128 <span itemprop="author" itemscope itemtype="http://schema.org/Person"> 129 by <span itemprop="name"></span> 130 </span> 131 132 <span> 133 134 </span> 135 <!-- 136 <span class="share-buttons"> 137 <span class="share-button"><a class="twitter-share-button" href="https://twitter.com/share" data-show-count="false">Tweet</a>
137<script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
137</span> 138 139 <span class="share-button"><span class="fb-like" data-href="/2020/04/19/SAC-analysis.html" data-layout="button_count" data-action="like" data-size="small" data-show-faces="false" data-share="true"></span></span> 140 </span> 141 <div style="clear: both;"/> 142 --> 143 144 </p> 145 </header> 146 147 <div class="post-content" itemprop="articleBody"> 148 <blockquote> 149 <p>Off Policy Trainì íµí´, Sample Efficiency를 ëì¬ë³´ì.</p> 150</blockquote> 151 152<h1 id="soft-actor-critic-ë¶ì">Soft Actor Critic ë¶ì</h1> 153 154<blockquote> 155 <ul> 156 <li>Soft Actor Criticì Policy Gradient ë°©ìì¸ Actor Critic 모ë¸ì ë³íì´ë©´ìë Replay Bufferë¡ trainíë off policy 모ë¸ì´ë¤. PGì´ë©´ìë off policyê° ê°ë¥í ê²ì Soft Q-Learningì 기ë°ì¼ë¡ í기 ë문ì´ë¤.</li> 157 <li>SACë Continuous Action Space를 ëìì¼ë¡ íë 모ë¸ì´ë¤. SAC를 Discrete Action Spaceì ì ì©í ì ìëë¡ ìì í 모ë¸ë ìë¤.</li> 158 <li>ë ¼ë¬¸ììë 4족보íë¡ë´(locomotion for a quadrupedal robot)ìì SOTA ì±ë¥ì´ ëì¨ë¤ê³ ì¸ê¸íê³ ìë¤. ì ë°ì ì¸ taskììë PPOë³´ë¤ ëª»íë¤.</li> 159 </ul> 160</blockquote> 161 162<hr /> 163 164<h2 id="ê°ì">ê°ì</h2> 165 166<ul> 167 <li>2018ë 1ì, <a href="https://arxiv.org/abs/1801.01290" target="_blank">Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, Sergey Levine. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor</a></li> 168 <li>2018ë 12ì <a href="https://arxiv.org/abs/1812.05905" target="_blank">Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, Sergey Levine. Soft Actor-Critic Algorithms and Applications</a>: Value Networkì ìì ê³ , Q-Networkê³¼ Policy Networkì¼ë¡ 구ì±. Policy Neworkì ì¬ì©íë temperature hyperparameter \(\alpha\)를 trainíë ìì´ ì¶ê°ëìë¤.</li> 169 <li>SAC for Discrete Action Space(2019ë 10ì): <a href="https://arxiv.org/abs/1910.07207" target="_blank">Soft Actor-Critic for Discrete Action Settings</a></li> 170 <li>Soft Q-Learning(2017ë 2ì): <a href="https://arxiv.org/abs/1702.08165" target="_blank">Reinforcement Learning with Deep Energy-Based Policies</a></li> 171</ul> 172 173<table> 174 <thead> 175 <tr> 176 <th style="text-align: center">PPO</th> 177 <th style="text-align: center">SAC</th> 178 </tr> 179 </thead> 180 <tbody> 181 <tr> 182 <td style="text-align: center"><img src="/assets/images/pendulum-ppo.gif" alt="" style="width: 70%;" class="center" /></td> 183 <td style="text-align: center"><img src="/assets/images/pendulum-sac.gif" alt="" style="width: 70%;" class="center" /></td> 184 </tr> 185 </tbody> 186</table> 187 188<p>*그림: PPOê° SACë³´ë¤ ìì ì ì´ë¤. SACëª¨ë¸ ê²°ê³¼ê° ì¢ ë ë¶ìì í´ ë³´ì´ë ì´ì ë, SAC 모ë¸ì´ ê°ë¥í random actionì ì¶êµ¬íë©´ìë ì무(task)를 ì±ê³µìí¤ë ê²ì´ 목ì ì´ê¸° ë문ì´ë¤.</p> 189 190<h2 id="policy--entropy">Policy & Entropy</h2> 191 192<ul> 193 <li>Policy Gradientì ì¼ë°ì ì¸ ì´ì¼ê¸°ë¥¼ í´ë³´ì. Policy Gradientì Objective Functionì ë¤ìê³¼ ê°ì ííë¡ ì£¼ì´ì§ë¤. íë¥ \(p\)ì¸ actionì ì·¨í´ reward \(r\)ì ë°ì ìí©ì´ë¤.</li> 194</ul> 195 196<p>\(\max\ [r\log p] \quad \quad \text{or} \quad \quad \max\ [A \log p]\)</p> 197<ul> 198 <li></li> 199 <li> 200 <p>Policy ìµì íë ëì reward \(r\)ì 주ë í룰 \(p\)를 ë í¬ê² íê±°ë ëì advantage \(A\)를 주ë íë¥ \(p\)를 ë í¬ê² íë ¤ê³ ìµì í를 ìííë¤. (\(r,p\))ì ìì´ ë¬¶ì¬ ì기 ë문ì on policyê° ëë¤. gradient update(train)ì íµí´ networkì´ ë³íë¤ë©´, networkì´ ìì±íë ìë¡ì´ íë¥ ì reward \(r\)ì ë§ë¤ì´ë¸ íë¥ ì´ ìëë¤. ì´ë¬í ì´ì ë¡ off policyê° ë ì ìë¤.</p> 201 </li> 202 <li>SACììë Objective Functionì´ ë¤ìê³¼ ê°ì´ 주ì´ì§ë¤.</li> 203</ul> 204 205\[\max\ \big[Q + \alpha (-\log p)\big] \quad \quad \text{or} \quad \quad \min\ \big[\alpha \log p - Q\big]\] 206 207<ul> 208 <li>ì´ ìì 2ê°ì§ë¡ í´ìí ì ìë¤. 209 <ul> 210 <li>í´ì1: target íë¥ ì \(\frac{1}{\alpha}\exp(Q)\)(ëë softmax \(Q\))ë¡ ë³´ê³ , íë¥ \(p\)를 ìì±í distribution \(\pi\)ì \(\frac{1}{\alpha}\exp Q\)ìì KL-Divergence를 ìµìí íë¤. ì¦ \(D_{KL}\big(\pi \ \Vert\ \frac{1}{\alpha}\exp Q\big) \approx \alpha \log p - Q\)</li> 211 <li>í´ì2: \(Q\)ê°ê³¼ entorpy \((-\log p)\)ì weighted sumì ìµëííë¤. entropyì weightê° \(\alph
211a\)ì´ë¤. \(Q\)를 advantageë¡ ëì²´íë ê²ë ê°ë¥íë¤.</li> 212 </ul> 213 </li> 214 <li>SACë ë³´íµì Policy Gradientì ë¤ë¥¸ Soft Q-Learningì ì¬ì©í기 ë문ì, Replay Buffer를 ì¬ì©íë Off Policy 모ë¸ì´ë¤. Soft Q-Learningì \(\exp Q\) (ëë softmax \(Q\)) ê°ì policy íë¥ ë¡ ë³´ë ë°©ë²ì´ë¤.</li> 215</ul> 216 217<p><img src="/assets/images/sac-unimoda-multimodall-policy.png" alt="" style="width: 100%;" class="center" /></p> 218 219<p>*그림: A multimodal \(Q\)-function: 1. ì¼ìª½ 그림과 ê°ì \(Q\)-functionì´ ì£¼ì´ì ¸ ìì ë, policy를 ì ê·ë¶í¬ë¡ ì측íë¤ë©´, ì¤ë¥¸ìª½ì ìì modeì ëí action íë¥ ì´ ëìì§ ì ìë¤. 2. policy를 \(\exp Q(s_t,a_t)\)ì ë¹ë¡íê² ë§ë ë¤ë©´, ìí© ë³íì ë ì ëìí ì ìë 모ë¸ì ë§ë¤ ì ìë¤.</p> 220 221<ul> 222 <li>SAC모ë¸ì ì¼ë°ì ì¸ sate value functionì ëì íì¬, <u>soft state value function</u>ì ë¤ìê³¼ ê°ì´ ì ìíë¤. ì¼ë°ì ì¸ state value functionì etropy termì ì¶ê°íì¬ explorationì´ ê°ë¥íëë¡ íë¤.</li> 223</ul> 224 225\[\begin{eqnarray} 226H(p) &=& \mathbb{E}_{x\sim p} \Big[ -\log p(x) \Big], \nonumber\\ 227 V(s_t) &=& \mathbb{E}_{a_t\sim\pi} \big[ Q(s_t,a_t) \big] +\alpha H \big(\pi(\cdot | s_t)\big) \nonumber\\ 228 &=& \mathbb{E}_{a_t\sim\pi} \big[ Q(s_t,a_t) -\alpha \log \pi(a_t \vert s_t) \big] \ \ \text{for } \alpha > 0, \label{eq43}\tag{1}\\ 229Q(s_t,a_t) &=& \mathbb{E}_{s_{t+1}\sim p, a_{t+1} \sim \pi} \Bigg[ r(s_t, a_t, s_{t+1}) + \gamma \Big( Q(s_{t+1}, a_{t+1}) + \alpha H(\pi(\cdot | s_{t+1})) \Big) \Bigg]\nonumber \\ 230&=& \mathbb{E}_{s_{t+1}\sim p} \Bigg[ r(s_t, a_t, s_{t+1}) + \gamma V(s_{t+1}) \Bigg] \nonumber 231\end{eqnarray}\] 232 233<ul> 234 <li>entropy regularization coefficient(or temperature coefficient) \(\alpha\)ë ììë¡ ê³ ì íë ê²½ì°ë ìê³ , trainingì íµí´ updateíë ê²½ì°ë ìë¤.</li> 235 <li> 236 <p>ì´ë¡ ì ì¼ë¡ ìì ê°ì´ soft value functionì ì ìí´ë optimal policyë¡ ìë ´íë¤.</p> 237 </li> 238 <li>Policy Network(\(\pi_\phi\)), Value Network(\(V_\psi, V_{\bar{\psi}}\)), Q-Network(\(Q_{\theta}\))ì´ íìíë¤. Valueê°ì \(Q\)ê°ì¼ë¡ ë¶í° 구í ìë ìì§ë§, SACììë 모ë¸ì ìì ì±ì ìí´ ë³ëì Value Networkì ëìë¤. \(V_{\bar{\psi}}\)ë Value target Networkì¼ë¡ train ëìì´ ëì§ë ìê³ , \(V_\psi\)ë¡ ë¶í° exponentially moving averageë¡ updateëë¤. ëë²ì§¸ SAC ë ¼ë¬¸ììë Value Network ìì´ Q-Networkë§ì¼ë¡ 구ííê³ ìë¤. continuous action spaceì´ê¸° ë문ì, Q-Networkì ì ë ¥ì stateì actionì concatì´ ëë¤.</li> 239 <li>SACë Replay Buffer를 ì¬ì©íë off policy 모ë¸ì´ë¤. Replay Buffer \(\mathcal{D} = \{ (s_t,a_t,r_t,d_n,s_{t+1}) \}\)ë¡ êµ¬ì±ëë¤.</li> 240 <li>Value Loss: Value Network \(V_\psi\) training.</li> 241</ul> 242 243\[\begin{eqnarray} 244J_V(\psi) = \mathbb{E}_{s_t \sim \mathcal{D}}\Bigg[{\frac{1}{2}\bigg(V_\psi(s_t) - \mathbb{E}_{\bar{a}_t\sim\pi_\phi}\big[{Q_\theta(s_t, \bar{a}_t) - \alpha\log \pi_\phi(\bar{a}_t|s_t)}\big] \bigg)^2} \Bigg] \label{eq41}\tag{2} 245\end{eqnarray}\] 246 247<p><img src="/assets/images/sac_1.png" alt="" style="width: 70%;" class="center" /></p> 248 249<p>*그림: Value Loss: \(\bar{a}_t\)ë replay bufferììì actionì´ ìëê³ , policy networkìì ìë¡ê² ìì±í actionì´ë¤. ì°¸ê³ ë¡ \(s_t\) ëì \(s_{t+1}\)를 ì´ì©í´ì value loss를 구í ìë ìë¤.</p> 250 251<ul> 252 <li>ì(\(\ref{eq41}\))ìì 기ëê°ì ì§ì ê³ì°í기 ì´ë µê¸° ë문ì policy network \(\pi_\phi\)ìì samplingì íµí´ action \(\bar{a}_t\)를 ë½ìì ê³ì°íë©´ ëë¤.</li> 253</ul> 254 255\[\mathbb{E}_{\bar{a}_t\sim\pi_\phi}\big[Q_\theta(s_t, \bar{a}_t) - \alpha\log \pi_\phi(\bar{a}_t|s_t)\big] \ \rightarrow \ Q_\theta(s_t, \bar{a}_t) - \alpha\log \pi_\phi(\bar{a}_t|s_t)\] 256 257<ul> 258 <li> 259 <p>Q-Function Loss: Q-Network(\(Q_{\theta}\)) training. 260\(\begin{eqnarray} 261J_Q(\theta) &=& \mathbb{E}_{(s_t,a_t)\sim\mathcal{D}} \Bigg[ \frac{1}{2} \bigg( Q_\theta(s_t,a_t) - \underbrace{\hat{Q}(s_t,a_t)}_{\text{stop gradient}} \bigg)^2 \Bigg] \ \ \text{with} \nonumber \\ 262\hat{Q}(s_t,a_t) &:=& r(s_t,a_t) + \gamma \mathbb{E}_{s_{t+1}\sim p} \Big [ V_{\bar{\psi}} (s_{t+1}) \Big] \ \ \leftarrow {p: \text{transition probability.}} \label{eq42}\tag{3} 263\end{eqnarray}\)</p> 264 </li> 265 <li>next state \(s_{t+1}\)ì \(s_{t+1}\sim p\)ë¡ íìëì´ ìì§ë§, 구íììë replay bufferì ìë \(s_{t+1}\)ì´ë¤. \(\hat{Q}(s_t,a_t)\)ì gradientë ê³ì°ëì§ ììì¼ íë¤.</li> 266 <li>ì(\(\ref{eq41}\))ìì Value Networkì targetì Q-Networkê°ì´ ì¬ì©ëê³ , ì(\ref{eq42})ìì Q-Newtorkì targetì Value Networkì´ ì¬ì©ëë¤. ì´ë° ìí 참조 구조를 ë²ì´ë기 ìí´, Value Networkì copyì¸ Value target Network \(V_{\bar{\psi}}\)를 ë§ëë ê²ì´ë¤. 267ë ë¤ë¥¸ 측면ì¼ë¡, ì(\ref{eq42})ìì Value Network \(V_{\psi}\) ëì target Network \(V_{\bar{\psi}}\)를 ì¬ì©íë ê²ì DQNìì main network, target networkì¼ë¡ 2ê° ì¬ì©íë ê²ê³¼ ê°ì ì리ì´ë¤.</li> 268 <li>target value Network \(V_{\bar{\psi}}\) update ìí¤ë ë°©ì 2ê°ì§. 269 <ul> 270 <li>주기ì ì¼ë¡(periodically) update.</li> 271 <li>Polyak averaging: exponentially weighted averaging</li> 272 </ul> 273 </li> 274 <li>Policy Loss: expected KL-Divergence</li> 275</ul> 276 277\[J_{\pi}(\phi) = \mathbb{E}_{s_t \sim \mathcal{D}} \Bigg [ D_{KL} \bigg( \pi_{\phi}
277 (\cdot | s_t)\ \Vert\ \frac{\exp(\frac{1}{\alpha}Q_\theta(s_t, \cdot))}{Z_\theta(s_t)} \bigg) \Bigg]\] 278 279<ul> 280 <li>ë³´íµì Policy Gradient 모ë¸ì´ Off Policy ë°©ë²ì ì ì©í기 ìí´, Old Policyë¡ ìì±ë Reaplay Bufferì dataë¡ New Policy를 updateí기 ìí ë°©ë²ì´ ìì´ì¼ íë¤. TRPO, PPOììë probability ratio를 ì´ì©í important sampling 기ë²ì ì¬ì©íë¤. 281SACììë \(\exp Q(s_t,\cdot)\)를 normalization(\(Z_\theta(s_t)\))í´ì íë¥ ë¶í¬ë¡ ê°ì£¼íë¤. ê·¸ëì new policyì \(\exp Q(s_t, \cdot)\)ì KL-Divergenceê° ìµìí ëëë¡ íë¤. ì¤ì ê³ì°ììë normalization termì¸ \(Z_\theta(s_t)\)ë 무ìí´ë ëë¤.</li> 282</ul> 283 284\[\begin{eqnarray} 285J_{\pi}(\phi) &=& \mathbb{E}_{s_t \sim \mathcal{D}} \Bigg [ \mathbb{E}_{\bar{a}_t\sim \pi_{\phi}} \Big[ \alpha \log \pi_{\phi} ( \bar{a}_t | s_t) - Q_\theta(s_t, \bar{a}_t) \Big] \Bigg] \label{eq44}\tag{4} \\ 286&\approx& \mathbb{E}_{s_t \sim \mathcal{D}} \Big [ \alpha \log \pi_{\phi} ( \bar{a}_t | s_t) - Q_\theta(s_t, \bar{a}_t) \Big] \label{eq49} \tag{5} 287\end{eqnarray}\] 288 289<ul> 290 <li>ì´ ìììë 기ëê° ê³ì°ì Value Lossì ëì¼íê² samplingì íµí´ ì²ë¦¬íë¤. ê·¸ë°ë°, samplingë \(\bar{a}_t\)ì ê´í backpropagationì 미ë¶ì´ íìí기 ë문ì Policy Networkì distributionìì reparameterization trickì´ íìíë¤.</li> 291 <li>ì(\ref{eq43}), (\ref{eq44})를 ë¹êµí´ ë³´ì. KL-Divergence를 ìµìí íë¤ë ê²ê³¼ soft value function ê°ì ìµëí íë¤ë ê²ì´ ëì¹ë¼ë ê²ì ì ì ìë¤.</li> 292 <li>2ê°ì \(Q\)-Networks: Our algorithm also makes use of two Q-functions to mitigate positive bias in the policy improvement step that is known to degrade performance of value based methods. \(Q_{\theta}\) ëì , 2ê°ì \(Q_{\theta_1}, Q_{\theta_2}\)ì ê°ê° ë 립ì ì¼ë¡ train ìì¼ì minê°ì ì ì©íë¤. ì´ë° ë°©ë²ì train ìë í¥ììë ëìì´ ëë¤. ì(\ref{eq41}), (\ref{eq44})ìì \(Q_\theta\)를 \(\min \big[ Q_{\theta_1}, Q_{\theta_2} \big]\)ì¼ë¡ ë°ê¾¸ë©´ ëë¤.</li> 293 <li>squashing function \(\tanh\): ë³´íµ continuous action spaceìì policy networkì ì ê·ë¶í¬ë¡ 모ë¸ë§íê³ action range를 ë²ì´ëë actionì clippingì¼ë¡ ì²ë¦¬íë¤. SACë ¼ë¬¸ììë \(u\sim N(\mu,\sigma)\)를 ì측íê³ \(a=\tanh(u)\)ë¡ actionì ì측íë¤. ë°ë¼ì action \(a\)ì likelihood(probability density function(pdf)ê°)ì change of variables를 ì´ì©í´ì ê³ì°í´ì£¼ë©´ ëë¤.</li> 294 <li>먼ì ì¼ë°ì ì¸ íë¥ ë³ìì íë¥ ë°ë í¨ìììì change of variablesì ëí´ ì´í´ë³´ì. íë¥ ë³ì \(X\)ì pdfê° \(f_X(x)\)ë¼ íì. ì´ë íë¥ ë³ì \(Y:=g(X)\)ì pdf를 구íë ê³¼ì ì´ change of variablesì´ë¤. íë¥ ë³ì \(Y\)ì pdf \(f_Y(y)\)ë ë¤ìê³¼ ê°ì´ 주ì´ì§ë¤.</li> 295</ul> 296 297\[\begin{eqnarray*} 298 f_Y(y) &=& f_X\big(g^{-1}(y)\big) \overbrace{\left| \frac{d}{dy} g^{-1}(y) \right|}^{\text{Jacobian determinantì ì ëê°}} \\ 299&=& f_X(x) \left| \frac{d}{dy} g^{-1}(y) \right| 300\end{eqnarray*}\] 301 302<ul> 303 <li>\(u\)ì pdf를 \(\mu(u\vert s)\)ë¼ íì. ì°ë¦¬ì ê²½ì°ë \(u=\tanh^{-1}(a)\)ì jacobian(\(\frac{du}{da}\))ì´ ëê°íë ¬ì´ê¸° ë문ì, \(\tanh\)ì ìí¨ì를 미ë¶í´ì¼ íë \(\frac{du}{da}\)ë³´ë¤ë \(\tanh\)를 ë°ë¡ 미ë¶íë \(\frac{da}{du}\) 미ë¶í´ì ìì를 ì·¨íë ê²ì´ ë«ë¤.</li> 304</ul> 305 306\[\begin{eqnarray*} 307\frac{da}{du} &=& \text{diag}(1-\tanh^2(u)) \\ 308\pi(a\vert s) &=& \mu(u\vert s) \left| \det\left( \frac{du}{da} \right) \right| \\ 309 &=& \mu(u\vert s) \left| \det\left( \frac{da}{du} \right) \right|^{-1} \\ 310\log \pi(a\vert s) &=& \log \mu(u\vert s) - \sum_i \log \big( 1-\tanh^2(u_i) \big) 311\end{eqnarray*}\] 312 313<ul> 314 <li>ì¬ê¸°ì \(\mu(u\vert s)\)ì ê° action ì±ë¶ì´ uncorrelatedëì´ ìë¤ë©´, íë¥ ì ê° action ì±ë¶ íë¥ ì ê³±ì´ ëë¤. ë°ë¬ì,</li> 315</ul> 316 317\[\log \mu(u\vert s) = \sum _i \log \mu(u_i \vert s)\] 318 319<h2 id="ëë²ì§¸-ë ¼ë¬¸-value-network-ìì´-q-networkë§ì¼ë¡-모ë¸-구ì
319±">ëë²ì§¸ ë ¼ë¬¸: Value Network ìì´ Q-Networkë§ì¼ë¡ ëª¨ë¸ êµ¬ì±</h2> 320 321<ul> 322 <li>Value Networkìì´ Q-Networkê³¼ Policy Networkì¼ë¡ë§ 구ì±. Q-Networkì´ \(Q_{\theta_1}, Q_{\theta_2}\) 2ê° ìì´ì¼ íê³ , target Q-Network \(Q_{\bar{\theta}_1}, Q_{\bar{\theta}_2}\)ë ìì´ì¼ í기 ë문ì, 모ë 4ê°ì Q-Networkì´ íìíë¤.</li> 323 <li>Value Networkì´ ì¬ì©ëë ì(\ref{eq42})를 ìì í´ì¼ íë¤. Replay Buffer \(\mathcal{D}\)ë¡ ë¶í° data \((s_t,a_t,r_t,d_n,s_{t+1})\)ê° ì£¼ì´ì ¸ ìë¤ë©´, Policy Networkì \(s_{t+1}\)ì ë£ì´ samplingíë©´ \(\bar{a}_{t+1}, \pi(\bar{a}_{t+1} \vert s_{t+1})\)를 ì»ì ì ìë¤.</li> 324</ul> 325 326\[\begin{eqnarray} 327 J_Q(\theta_i) &=& \mathbb{E}_{(s_t,a_t)\sim\mathcal{D}} \Bigg[ \frac{1}{2} \bigg( Q_{\theta_i}(s_t,a_t) - \underbrace{\hat{Q}(s_t,a_t)}_{\text{stop gradient}} \bigg)^2 \Bigg] \ \ \text{with} \nonumber \\ 328 \hat{\mathbb{Q}}(s_t,a_t) &=& r(s_t,a_t) + \gamma \mathbb{E}_{a_{t+1}\sim p}\Bigg(\min\Big [ Q_{\bar{\theta}_1}(s_{t+1},a_{t+1} ), Q_{\bar{\theta}_2}(s_{t+1},a_{t+1} ) \Big] - \alpha \log \pi(a_{t+1} \vert s_{t+1}) \Bigg) \label{eq45}\tag{6}\\ 329 &\approx& r(s_t,a_t) + \gamma \Bigg(\min\Big [ Q_{\bar{\theta}_1}(s_{t+1},\bar{a}_{t+1} ), Q_{\bar{\theta}_2}(s_{t+1},\bar{a}_{t+1} ) \Big] - \alpha \log \pi(\bar{a}_{t+1} \vert s_{t+1}) \Bigg) \label{eq46}\tag{7} 330\end{eqnarray}\] 331 332<ul> 333 <li>ì(\ref{eq45})ì ê°ì´ state \(s_{t+1}\)ììì 기ëê°ì ê³ì°í´ì¼ íì§ë§, 기ëê°ì ê³ì°í기 ì´ë ¤ì°ë¯ë¡ ëì í¹ì action \(\bar{a}_{t+1}\)ì íë ìì±í´ì ì(\ref{eq46})ê³¼ ê°ì´ ì²ë¦¬íë¤. ë§ì½ discrete action spaceìë¤ë©´, ê·¸ 기ëê°ì ê³ì°íê¸°ê° ì½ë¤.</li> 334</ul> 335 336<h2 id="temperature-hyperparameter-alpha-train-model">Temperature Hyperparameter \(\alpha\) Train Model</h2> 337<ul> 338 <li>entropy regularization coefficient(or temperature coefficient) \(\alpha\)를 hyperparameterë¡ ê³ ì íì§ ìê³ , trainì íµí´ ì°¾ë ë°©ë²ì ìê°í´ ë³´ì. minimum policy entropy thresholdì¸ ìì \(\mathcal{H}_0\)ì ëíì¬, entropyê° \(\mathcal{H}_0\)ë³´ë¤ ì»¤ì§ëë¡ ì ì½ì ëë¤ê³ í´ë³´ì</li> 339</ul> 340 341\[\begin{eqnarray*} 342& &\max_{\pi_0, \dots, \pi_T} \mathbb{E} \Bigg[ \sum_{t=0}^T r(s_t, a_t)\Bigg] \\ 343\text{s.t. } & & \mathbb{E}\Big[-\log \pi_t(a_t|s_t) \Big] \geq \mathcal{H}_0, \ \ \forall t 344\end{eqnarray*}\] 345 346<ul> 347 <li>ì´ ë¬¸ì 를 dual problemì¼ë¡ ë³í í, time step \(t=T, T-1, T-2, \cdots\) ììì backwardë¡ ì ê°íì¬ ìµì¢ ì ì¸ minimization 문ì ì objective functionì ì»ê² ëë¤.</li> 348</ul> 349 350\[\begin{eqnarray} 351J(\alpha) &=& \mathbb{E}_{a_t\sim \pi_t} \Big[ -\alpha \log \pi_t(a_t \vert s_t) - \alpha \mathcal{H}_0 \Big] \label{eq47}\tag{8}\\ 352 &=& \alpha \overbrace{\mathbb{E}_{a_t\sim \pi_t} \Big[ - \log \pi_t(a_t \vert s_t) - \mathcal{H}_0 \Big]}^{\text{stop gradient}}\\ 353&\approx& -\alpha \log \pi_t(\bar{a}_t \vert s_t) - \alpha \mathcal{H}_0 \label{eq48}\tag{9} 354\end{eqnarray}\] 355 356<ul> 357 <li>ìì¸í ì ê° ê³¼ì ì <a href="https://lilianweng.github.io/lil-log/2018/04/08/policy-gradient-algorithms.html#sac-with-automatically-adjusted-temperature" target="_blank">ì´ê³³</a>ì ì°¸ê³ íë©´ ëë¤. dual problemì¼ë¡ ë³íí´ì ì ê°í í, ì¤ì§ì ì¼ë¡ \(\alpha\)를 구íë networkê³¼ policy를 구íë networkì´ ë¶ë¦¬ëì´ êµ¬í´ì ¸ì¼ í기 ë문ì, \(Q_t, \pi_t\)를 구í í, \(\alpha_t\)를 구íê² ëë¤. ê·¸ëì \(Q, \pi\)ì ë¶ë¦¬ë \(\alpha\)ë§ì optimization 문ì ê° ë§ë¤ì´ì§ë¤. ì¦ \(J(\alpha)\)를 ìµìí íë©´ ëë¤. \(J(\alpha)\)ë 기ëê°ì¼ë¡ ííë ìì¸ë°, 구체ì ì¸ ê¸°ëê°ì 구í기 ì´ë µê¸° ë문ì, ì(\ref{eq46})ì ê°ì ë°©ìì¼ë¡ Policy Networkì \(s_t\)를 ë£ì´ action \(\bar{a}_t\)를 íë samplingí´ì ì(\ref{eq48})ê³¼ ê°ì´ ëì²´í ì ìë¤.</li> 358 <li>ì´ ìì ì§ê´ì ì¸ ì미ë ë¨ìíë¤. 359 <ul> 360 <li>entropy \(-\log \pi_t(a_t \vert s_t)\)ê° \(\mathcal{H}_0\)ë³´ë¤ í¬ë©´, minimizationì ìí´ì \(\alpha\)ê°ì ì¤ì´ê² ëë¤. ì¦ entropyê° ë무 í¬ë©´ ì(\ref{eq43}), (\ref{eq44})ìì \(\alpha\)ê°ì 0ì ê°ê¹ê² ë§ë ë¤.</li> 361 <li>ë°ëë¡, entropyê° \(\mathcal{H}
361_0\)ë³´ë¤ ìì¼ë©´, \(\alpha\)ê°ì´ 커ì§ê² ëë¤.</li> 362 <li>구íììë target entropy \(\mathcal{H}_0\)를 0ë³´ë¤ ë®ê² ì¡ìëê³ , \(\alpha\)ê°ì´ trainì í ìë¡ ê³ì ë®ìì§ê² íë¤. \(\alpha\)ì ì´ê¸°ê°ê³¼ leaning rateì ì ì¤ì í´ì¼ íë¤.</li> 363 </ul> 364 </li> 365 <li>ì°¸ê³ ë¡ \(t=T\)ì¸ ê²½ì°ì ëí´ìë§, dual problemì ì´í´ë³´ì. dual variable \(\alpha_T \geq 0, -\log\pi_T(a_T \vert s_T) \geq \mathcal{H}_0\)ì ëíì¬ ë¤ì ìì´ ì±ë¦½íë¤.</li> 366</ul> 367 368\[A_T: = \max_{\pi_T} \mathbb{E}_{s_T, a_T} \Big [ r(s_T, a_T) \Big] \leq \max _{\pi_T} \mathbb{E} \Big[ r(s_T, a_T) + \alpha_T\big( -\log\pi_T(a_T \vert s_T) - \mathcal{H}_0 \big) \Big] \ \longleftarrow \text{constraint ì¡°ê±´ íìì}\] 369 370<ul> 371 <li>ì¬ê¸°ì constraint \(-\log\pi_T(a_T \vert s_T) \geq \mathcal{H}_0\) ì¡°ê±´ì ì ê±°(\(\alpha_T \geq 0\)ì ì ì§)íë©´ ë ë¤ìí ê°ì ê°ì§ ì ìì¼ë¯ë¡ maxê°ì´ ë ì»¤ì§ ì ìë¤. ë°ë¼ì, ë¤ì ìì´ ì±ë¦½íë¤(weak duality).</li> 372</ul> 373 374\[\overbrace{A_T}^{\text{constraintê° ì ì©ë ìµëê°}} \leq \quad \quad \max _{\pi_T} \mathbb{E} \Big[ r(s_T, a_T) + \alpha_T\big( -\log\pi_T(a_T \vert s_T) - \mathcal{H}_0 \big) \Big]\] 375 376<ul> 377 <li>ì´ ìì 모ë \(\alpha_T \geq 0\)ì ëíì¬ ì±ë¦½íë¯ë¡, minimumì ì·¨í´ë ì±ë¦½íë¤.</li> 378</ul> 379 380\[A_T \quad \leq \quad \min_{\alpha_T \geq 0} \max _{\pi_T} \mathbb{E} \Big[ r(s_T, a_T) + \alpha_T\big( -\log\pi_T(a_T \vert s_T) - \mathcal{H}_0 \big) \Big]\] 381 382<ul> 383 <li>ê·¸ë°ë° ì´ ë¶ë±ìì ë±ìì¼ë¡ ì±ë¦½íë¤. ì´ì ë Slaterâs conditionì´ ë§ì¡±ëì´, strong dualityê° ì±ë¦½í기 ë문ì´ë¤(the objective is linear and the constraint (entropy) is convex function in \(\pi_T\)).</li> 384</ul> 385 386\[A_T = \min_{\alpha_T \geq 0} \max _{\pi_T} \mathbb{E} \Big[ r(s_T, a_T) + \alpha_T\big( -\log\pi_T(a_T \vert s_T) - \mathcal{H}_0 \big) \Big]\] 387 388<ul> 389 <li>ì´ ìì ì§ê´ì ì¼ë¡ ì´í´í´ ë³¼ ì ìë¤. constraint ì¦ \(-\log\pi_T(a_T \vert s_T) \geq \mathcal{H}_0\) ê° ë§ì¡±ëì§ ìì¼ë©´, ìì \(\alpha_T\)를 무íí í¤ìì ì ì²´ ê°ì´ \(-\infty\)ë¡ ê°ê² ëëë°, ì´ë ê² í´ìë ìíë ìµëíê° ëì§ ìëë¤. ë°ë¼ì, ìµëí를 ì´ë£¨ê¸° ìí´ìë constraintê° ë§ì¡±ë ì ë°ì ìë¤.</li> 390</ul> 391 392<h2 id="soft-actor-critic-with-discrete-action-space">Soft Actor Critic with Discrete Action Space</h2> 393<ul> 394 <li>Discrete Action Spaceììë samplingì íµí´ ê·¼ì¬ì ì¼ë¡ 기ëê°ì ê³ì°íì§ ìê³ , 기ëê°ì ì§ì ê³ì°í ì ìë¤. ì(\ref{eq49}), (\ref{eq46}), (\ref{eq48})ì ìëëë¡ ì(\ref{eq44}), (\ref{eq45}), (\ref{eq47})ë¡ ê³ì°í ì ìë¤. ë°ë¼ì, samplingì´ íììê³ , replay bufferì ìë dataë§ ì¬ì©íë©´ ëë¤.</li> 395 <li>Discrete Soft Actor Critic 구íì DQN 구íê³¼ ì ì¬í ì¸¡ë©´ì´ ë§ë¤.</li> 396</ul> 397 398<table> 399 <thead> 400 <tr> 401 <th> </th> 402 <th style="text-align: center">Policy Network</th> 403 <th style="text-align: center">main Q-Network</th> 404 <th style="text-align: center">target Q-Network</th> 405 <th style="text-align: center"> </th> 406 </tr> 407 </thead> 408 <tbody> 409 <tr> 410 <td>Policy Loss</td> 411 <td style="text-align: center">\(\pi_\phi(s_t)\)</td> 412 <td style="text-align: center">\(Q_{\theta_i}(s_t)\)</td> 413 <td style="text-align: center"> </td> 414 <td style="text-align: center">ì(\ref{eq44})</td> 415 </tr> 416 <tr> 417 <td>\(Q\)-Loss</td> 418 <td style="text-align: center">\(\pi_\phi(s_{t+1})\)</td> 419 <td style="text-align: center">\(Q_{\theta_i}(s_t,a_t)\) \(\leftarrow\) prediction</td> 420 <td style="text-align: center">\(Q_{\bar{\theta}_i}(s_{t+1})\)</td> 421 <td style="text-align: center">ì(\ref{eq45})</td> 422 </tr> 423 <tr> 424 <td>\(\alpha\)-Loss</td> 425 <td style="text-align: center">\(\pi_\phi(s_t)\)</td> 426 <td style="text-align: center"> </td> 427 <td style="text-align: center"> </td> 428 <td style="text-align: center">ì(\ref{eq47})</td> 429 </tr> 430 </tbody> 431</table> 432 433<p>í: Discrete Soft Actor Critic: Ploicy Networkì Policy Loss, \(Q\)-Loss를 ê³ì°í ëë tensorflow graphì reuseê° íìíë¤. main \(Q\)-Networkì \(Q_{\theta_i}(s_t)\)를 íë² ê³ì°í í, \(Q\)-Lossììë \(a_t\)ììì ê°ë§ indexingíë©´ ëë¤.</p> 434 435<h2 id="gym-games">Gym Games</h2> 436<ul> 437 <li>Discrete Action, CartPole-v1: vanilla Policy Gradient ì ëì ì±ë¥ì´ ëì¨ë¤.</li> 438 <li>Continous Action, Pendulum-v0: PPOë³´ë¤ train ìëê° ëë¦¬ê³ , ê²°ê³¼ë ì¢ ë¶ìì íë¤. ì´ì ë ìì gif 그림ì ì°¸ê³ íë©´ ëë¤.</li> 439 <li>BreakoutDeterministic-v4: trainì´ ì ëì§ ìëë¤.</li> 440 <li>BipedalWalker-v3: train ìëë PPOë³´ë¤ ë리ë¤. SACë¡ trainí 모ë¸ì´ ë 빨리 ëìê°ë¤.</li> 441</ul> 442 443<table> 444 <thead> 445 <tr> 446 <th style="text-align: center">PPO</th> 447 <th style="text-align: center">SAC</th> 448 </tr> 449 </thead> 450 <tbody> 451 <tr> 452 <td style="text-align: center"><img src="/assets/images/bipedalwalker-ppo.gif" alt="" style="width: 70%;" class="center" /></td> 453 <td style="text-align: center"><img src="/assets/images/bipedalwalker-sac.gif" alt="" style="width: 70%;" class="center" /></td> 454 </tr> 455 </tbody> 456</table> 457 458<h2 id="reference">Reference</h2> 459 460<ul> 461 <li><a href="https://drive.google.com/open?id=16olGwVvk_smtgopmuUtouOf1ad1RGpIf" target="_blank">ë¥ë¬ë ì 리 ìë£</a></li> 462 <li><a href="https://bair.berkeley.edu/blog/2017/10/06/soft-q-learning/" target="_blank">https://bair.berkeley.edu/blog/2017/10/06/soft-q-learning/</a></li> 463</ul> 464 465 </div> 466 467 468 <div class="page-navigation"> 469 470 <a class="prev" href="/2020/03/27/ACER-analysis.html">← ACER ë½ê°ê¸°</a> 471 472 473 474 </div> 475 476 477 <div id="disqus_thread"></div>
478<script> 479 480/** 481* RECOMMENDED CONFIGURATION VARIABLES: EDIT AND UNCOMMENT THE SECTION BELOW TO INSERT DYNAMIC VALUES FROM YOUR PLATFORM OR CMS. 482* LEARN WHY DEFINING THESE VARIABLES IS IMPORTANT: https://disqus.com/admin/universalcode/#configuration-variables*/ 483/* 484var disqus_config = function () { 485this.page.url = PAGE_URL; // Replace PAGE_URL with your page's canonical URL variable 486this.page.identifier = PAGE_IDENTIFIER; // Replace PAGE_IDENTIFIER with your page's unique identifier variable 487}; 488*/ 489(function() { // DON'T EDIT BELOW THIS LINE 490var d = document, s = d.createElement('script'); 491s.src = 'https://https-hccho2-github-io.disqus.com/embed.js'; 492s.setAttribute('data-timestamp', +new Date()); 493(d.head || d.body).appendChild(s); 494})(); 495</script>
495 496<noscript>Please enable JavaScript to view the <a href="https://disqus.com/?ref_noscript">comments powered by Disqus.</a></noscript> 497 498 499</article> 500 501 </div> 502 </main> 503 504 <div style="clear: both;"/> 505<footer class="site-footer"> 506 <p> 507 <a href="https://github.com/hccho2" target="_blank"> 508 <img src='/assets/images/logo_github.png' /> 509 </a> 510 </p> 511</footer> 512 513 514 </body> 515 516</html>
Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.