1<!DOCTYPE html> 2<html lang="en"><head> 3 <meta charset="utf-8"> 4 <meta http-equiv="X-UA-Compatible" content="IE=edge"> 5 <meta name="viewport" content="width=device-width, initial-scale=1"><!-- Begin Jekyll SEO tag v2.8.0 --> 6<title>Chad Austin</title> 7<meta name="generator" content="Jekyll v3.9.3" /> 8<meta property="og:title" content="Chad Austin" /> 9<meta property="og:locale" content="en_US" /> 10<link rel="canonical" href="https://chadaustin.me/" /> 11<meta property="og:url" content="https://chadaustin.me/" /> 12<meta property="og:site_name" content="Chad Austin" /> 13<meta property="og:type" content="website" /> 14<meta name="twitter:card" content="summary" /> 15<meta property="twitter:title" content="Chad Austin" />
16<script type="application/ld+json"> 17{"@context":"https://schema.org","@type":"WebSite","headline":"Chad Austin","name":"Chad Austin","url":"https://chadaustin.me/"}</script>
17 18<!-- End Jekyll SEO tag --> 19<link rel="stylesheet" href="/assets/main.css"><link type="application/atom+xml" rel="alternate" href="https://chadaustin.me/feed/atom" title="Chad Austin" /><!-- Global site tag (gtag.js) - Google Analytics -->
vendor: 64 bytes, lines 19-20
19 20<script async src="https://www.googletagmanager.com/gtag/js?id=
20UA-4177925-1
vendor: 12 bytes, line 20
20"></script>
21<script> 22
vendor: 133 bytes, lines 22-25
22window.dataLayer = window.dataLayer || []; 23 function gtag(){dataLayer.push(arguments);} 24 gtag('js', new Date()); 25 gtag('config', '
25UA-4177925-1
vendor: 4 bytes, line 25
25');
26</script>
26 27</head> 28<body class="home"><header class="site-header" role="banner"> 29 30 <div class="wrapper"><a class="site-title" rel="author" href="/">Chad Austin</a></div> 31</header> 32<main class="page-content" aria-label="Content"> 33 <div class="wrapper"> 34 <div class="home"> 35 36 37 <div class="home-content"> 38 <div class="recent-post-titles"> 39 <h1 class="post-title">Recent Posts</h1><div class="post-link-list"><div class="post-date">Mar 2025</div> 40 <a href="/2025/03/snes-classic-partial-repair/">(Partially) Repairing a Super NES Classic</a><div class="post-date">Oct 2024</div> 41 <a href="/2024/10/intrusive-linked-list-in-rust/">Unsafe Rust Is Harder Than C</a><div class="post-date">Feb 2024</div> 42 <a href="/2024/02/windows-terminal-latency/">Terminal Latency on Windows</a><div class="post-date">Feb 2024</div> 43 <a href="/2024/02/tmux-config/">My Minimal tmux Config</a><div class="post-date">Jan 2024</div> 44 <a href="/2024/01/truecolor-terminal-emacs/">I Just Wanted Emacs to Look Nice â Using 24-Bit Color in Terminals</a><div class="post-date">Nov 2023</div> 45 <a href="/2023/11/reference-counting-things/">Reference Counting Things</a><div class="post-date">Feb 2021</div> 46 <a href="/2021/02/wired-sculpt/">Microsoft Sculpt Wired Conversion Mod</a></div> 47</div> 48 <ul class="post-list"><li><span class="post-meta">Mar 20, 2025</span> 49 <h3> 50 <a class="post-link" href="/2025/03/snes-classic-partial-repair/"> 51 (Partially) Repairing a Super NES Classic 52 </a> 53 </h3> 54 <p>My friendâs SNES Classic stopped responding to controller inputs. He 55reset it to factory settings but couldnât even get through the 56language selection menu without being able to push buttons.</p> 57 58<p>I told him I could take a look.</p> 59 60<p>First I used Hakchi to save a backup of its internal storage.</p> 61 62<p>Then I flashed the factory kernel and system software and erased the 63user partition. Same thing. Controller did nothing.</p> 64 65<p>I tried my own controllers to no avail. And his controller worked in 66mine, so the issue was with the console itself.</p> 67 68<p>How do we debug further?</p> 69 70<p>The SNES Classic runs Linux on a somewhat mainstream ARM A7 SOC by 71Allwinner, and if you use Hakchi to flash its custom kernel, you can 72telnet to the device while itâs running and interrogate it.</p> 73 74<p>Unfortunately, I didnât keep great logs of this part of the process, 75but eventually I noticed something suspicious in <code class="language-plaintext highlighter-rouge">dmesg</code>:</p> 76 77<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[ -.------] twi_start()434 - [i2c1] START can't sendout! 78</code></pre></div></div> 79 80<p>That message comes from the Allwinner SOC i2c-sunxi kernel driver in 81<a href="https://github.com/linux-sunxi/linux-sunxi/blob/d47d367036be38c5180632ec8a3ad169a4593a88/drivers/i2c/busses/i2c-sunxi.c#L411">/drivers/i2c/busses/i2c-sunxi.c</a>.</p> 82 83<p>Oh no, an I²C start packet timeout sounds more like a hardware failure 84than anything my friend did with Hakchi. The SNES Classic 85controllers (and all Wii Nunchuk connectors) communicate with I²C (or 86TWI if you prefer that name). Itâs time to pop open the case.</p> 87 88<figure> 89<a href="/images/snes-classic/with-heat-sink.jpeg"><img src="/images/snes-classic/with-heat-sink.jpeg" alt="The inside of the case." /></a> 90<figcaption>The inside of the case.</figcaption> 91</figure> 92 93<p>With the case open and the heatsink off, thereâs not much to it.</p> 94 95<figure> 96<a href="/images/snes-classic/board-labeled.jpeg"><img src="/images/snes-classic/board-labeled.jpeg" alt="The unshielded board with labeled components." /></a> 97<figcaption>The unshielded board with labeled components.</figcaption> 98</figure> 99 100<p>The SOC is a four-core <a href="https://linux-sunxi.org/images/b/b3/R16_Datasheet_V1.4_(1).pdf">Allwinner 101R16</a> 102with a Mali GPU. Itâs quite a capable little chip.</p> 103 104<p>You can also see the DRAM (Nanya 105<a href="https://www.mxic.com.tw/Lists/Datasheet/Attachments/8462/MX30LF4G18AC,%203V,%204Gb,%20v1.4.pdf">NT5CC128M16IP-DI</a>), 106flash (Macronix 107<a href="https://www.mxic.com.tw/Lists/Datasheet/Attachments/8462/MX30LF4G18AC,%203V,%204Gb,%20v1.4.pdf">NT5CC128M16IP-DI</a>), 108and PMIC (X-Powers/Allwinner 109<a href="https://linux-sunxi.org/images/e/e5/AXP223_Datasheet_V1.0_en.pdf">AXP223</a>) 110on the top of the board. The HDMI circuitry (Explore 111<a href="https://bbs.aw-ol.com/assets/uploads/files/1639040585144-ep952-%E6%8A%80%E6%9C%AF%E5%8F%82%E6%95%B0.pdf">EP952</a>) 112is on the bottom.</p> 113 114<p>Most of thatâs irrelevant - the important part was whether I could 115find anything wrong with the I²C signals between the connector and the 116SOC.</p> 117 118<p>I wonât bore you with all of the random stuff I probed, but I 119eventually noticed that controller 1âs SCL line had a low-impedance 120path to ground, about 670 ohms. SDA and both of controller 2âs signal 121lines were pulled low at 1 MΩ, which makes a lot more sense.</p> 122 123<p>Having the clock incorrectly pulled low would explain why it couldnât 124communicate with controller 1. So where is the fault? Itâs either 125somewhere in the PCB (unlikely) or in the SOCâs I²C buffer.</p> 126 127<p>If the PCB, thatâs easily fixable: cut traces and bodge a wire from 128the closest good point. If itâs in the SOC, thatâs something I canât 129fix. I donât have the skill to reball and resolder BGA. While you can 130easily acquire an Allwinner R16 on Alibaba, thereâs no way I could 131reball and solder it to the board without breaking something else. I 132did find a service that does BGA rework but it starts at $200 and the 133SNES Classic isnât worth that much.</p> 134 135<p>It was a little hard to trace the path from the connectors to the SOC 136because the traces run on inner layers. Eventually, I found some 137jumpers to desolder to isolate the fault.</p> 138 139<figure> 140<a href="/images/snes-classic/i2c-labeled.jpeg"><img src="/images/snes-classic/i2c-labeled.jpeg" alt="Labeled I²C jumpers." /></a> 141<figcaption>Labeled I²C jumpers.</figcaption> 142</figure> 143 144<p>I desoldered SCL1âs jumper â well, letâs be honest. I melted that 145tiny sucker into a black paste while struggling to apply even heat. 146Unfortunately, SCL1âs short to ground is in the chip and I canât fix 147that.</p> 148 149<figure> 150<a href="/images/snes-classic/i2c-buffer-failure.png"><img src="/images/snes-classic/i2c-buffer-failure.png" alt="Likely a FET in the I²C buffer failed." /></a> 151<figcaption>Likely a FET in the I²C buffer failed.</figcaption> 152</figure> 153 154<p>
154At this point, software fixes became the only option.</p> 155 156<p>I was wondering if you could patch the Hakchi kernel to swap 157controller 1 with controller 2. It may not even be hard. Then at least 158one controller would function.</p> 159 160<p>But I also saw someone on Reddit mention that the system supports 161receiving power over USB while acting as a USB host with a powered 162<a href="https://en.wikipedia.org/wiki/USB_On-The-Go">USB On-The-Go</a> cable. 163You can build your own O2G cable with wire snips, a soldering iron, 164and heat shrink tubing. I just purchased <a href="https://www.amazon.com/dp/B00C452XFO">one from 165Amazon</a> instead. The reviews 166make it clear this is a popular purpose for the cable.</p> 167 168<figure> 169<a href="/images/snes-classic/usb-otg.jpeg"><img src="/images/snes-classic/usb-otg.jpeg" alt="Powered USB O2G Cable" /></a> 170<figcaption>Powered USB O2G Cable</figcaption> 171</figure> 172 173<p>The stock Nintendo kernel does not support USB controllers but the 174latest Hakchi kernel does! I confirmed an Xbox controller can navigate 175the menu and every button works in game. Even better, controller port 1762 still works as usual!</p> 177 178<p>I wonder if raphnetâs <a href="https://www.raphnet-tech.com/products/wusbmote_1player_adapter_v3/index.php">Classic to USB 179adapter</a> 180would work so you could keep the SNES controller experience.</p> 181 182<p>At this point, I declared as much victory as this was going to get and 183sent it back to my friend. Itâs nice to keep quality hardware out of 184the e-waste bin.</p> 185 186<figure> 187<a href="/images/snes-classic/it-works.jpeg"><img src="/images/snes-classic/it-works.jpeg" alt="It works!" /></a> 188<figcaption>It works!</figcaption> 189</figure> 190 191 192 </li><li><span class="post-meta">Oct 24, 2024</span> 193 <h3> 194 <a class="post-link" href="/2024/10/intrusive-linked-list-in-rust/"> 195 Unsafe Rust Is Harder Than C 196 </a> 197 </h3> 198 <h2 id="or-the-most-expensive-linked-list-ive-ever-written">Or: The Most Expensive Linked List Iâve Ever Written</h2> 199 200<p>Some of you already know the contents of this post, especially if 201youâve written embedded or unsafe code in Rust. But I didnât, so I 202thought it was useful to write down my experience as accurately as I 203can. Without further adoâ¦</p> 204 205<p>Last year, I wrote Photohash, <a href="https://github.com/chadaustin/photohash">software to help me index my NAS and 206find duplicate photos</a> with 207rotation-independent hashing and <a href="https://en.wikipedia.org/wiki/Perceptual_hashing">perceptual 208hashing</a>. To make 209use of cores and keep the disks busy, it distributes work to compute 210and IO workers. Work is distributed with channels â synchronized work 211queues.</p> 212 213<p>In Photohash, work tends to be discovered and processed in batches: 214enumerating directories returns multiple entries and the database is 215updated in multi-row transactions.</p> 216 217<p>Rust has a rich selection of channel implementations: 218<a href="https://doc.rust-lang.org/std/sync/mpsc/index.html">std::sync::mpsc</a>, 219<a href="https://docs.rs/futures/latest/futures/channel/index.html">futures::channel</a>, 220<a href="https://docs.rs/tokio/latest/tokio/sync/index.html">tokio::sync</a>, 221<a href="https://docs.rs/crossbeam/latest/crossbeam/channel/index.html">crossbeam::channel</a>, 222<a href="https://docs.rs/flume/">flume</a>, and <a href="https://docs.rs/kanal/">kanal</a> 223are high-quality options.</p> 224 225<p>Unfortunately, none of them exactly met my needs, so I nerd-sniped 226myself into writing my dream channel. My previous day job 227(<a href="https://github.com/facebook/sapling/tree/main/eden/fs">EdenFS</a> and 228<a href="https://github.com/facebook/watchman">Watchman</a>) was full of ad-hoc 229channels so I knew roughly I wanted. <code class="language-plaintext highlighter-rouge">kanal</code> is closest, but it is 230riddled with unsafe code and uses spinlocks which look great in 231microbenchmarks but have <a href="https://matklad.github.io/2020/01/02/spinlocks-considered-harmful.html">no place in userspace 232software</a>.</p> 233 234<p>Introducing 235<a href="https://docs.rs/batch-channel/">batch-channel</a>, 236a throughput-optimized channel. The design goals are:</p> 237 238<ul> 239 <li><strong>Multi-producer, multi-consumer</strong>. Parallelism in both production 240and consumption.</li> 241 <li><strong>Sync and async support</strong> for both consumers and producers. 242Mix-and-match provides flexibility for use in any type of thread 243pool, async runtime, or FFI.</li> 244 <li><strong>Bounded or unbounded</strong>. Bounded for backpressure and limiting peak 245memory consumption. Unbounded for situations where you cannot 246guarantee deadlock freedom.</li> 247 <li><strong>Sending and receiving multiple values</strong>. I often want to send 248multiple values. Like reading all of the paths in a directory. Or 249writing multiple rows to a database. Batching allows amortizing 250per-batch costs. Itâs silly to acquire the channel lock N times to 251push N values. Itâs the same on the consumption side: workers may 252want to pull all pending work items in one channel read. You might 253wonder about lock-free queues, and they have their place, but but 254youâll still contend on the head and tail, and atomic operations 255remain slow even on modern Intel cores. If youâre going to contend 256on the queue anyway, batch-channelâs philosophy is to stick the 257whole thing behind a mutex and maximize batch sizes on both ends.</li> 258</ul> 259 260<p>At the time this was written, the following design goals werenât yet 261implemented:</p> 262 263<ul> 264 <li><strong>Priorities</strong>. Senders can influence processing order.</li> 265 <li><strong>Bounding variable-sized items</strong>. For example, being able to say a 266queue can hold up to 20 MB of paths, no matter how long they are.</li> 267</ul> 268 269<p>And, finally, the design goal that led to this post:</p> 270 271<ul> 272 <li><strong>No allocations under steady-state use</strong>. Allocations are a source 273of contention, failure, and overhead, especially when using slow 274system allocators.</li> 275</ul> 276 277<h2 id="the-shape-of-a-channel">The Shape of a Channel</h2> 278 279<p>To explain why unsafe Rust is even involved, letâs look at the 280implementation of a channel.</p> 281 282<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">pub</span> <span class="k">struct</span> <span class="n">Channel</span><span class="o"><</span><span class="n">T</span><span class="o">></span> <span class="p">{</span> 283 <span class="n">q</span><span class="p">:</span> <span class="n">VecDeque</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">,</span> 284 <span class="c1">// Blocked receivers</span> 285 <span class="n">waiting_for_elements</span><span class="p">:</span> <span class="nb">Vec</span><span class="o"><</span><span class="n">
285Waker</span><span class="o">></span><span class="p">,</span> 286 <span class="c1">// Blocked senders</span> 287 <span class="n">waiting_for_capacity</span><span class="p">:</span> <span class="nb">Vec</span><span class="o"><</span><span class="n">Waker</span><span class="o">></span><span class="p">,</span> 288 <span class="c1">// Synchronous use requires some condition variables too.</span> 289<span class="p">}</span> 290</code></pre></div></div> 291 292<p>When an async receiver blocks on <code class="language-plaintext highlighter-rouge">recv()</code> because the channel is 293empty, the taskâs <code class="language-plaintext highlighter-rouge">Waker</code> is stored so that the channel knows to wake 294it when a value arrives.</p> 295 296<p><a href="https://doc.rust-lang.org/core/task/struct.Waker.html"><code class="language-plaintext highlighter-rouge">Waker</code></a> is a 297handle to a blocked task. The channel can signal the async runtime to 298wake a task when it should poll the channel again. Itâs a <a href="https://doc.rust-lang.org/nomicon/exotic-sizes.html">wide 299pointer</a>, two 300words in size.</p> 301 302<p><code class="language-plaintext highlighter-rouge">Waker</code> is used like this:</p> 303 304<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">pub</span> <span class="k">struct</span> <span class="n">Recv</span><span class="o"><</span><span class="nv">'a</span><span class="p">,</span> <span class="n">T</span><span class="o">></span> <span class="p">{</span> 305 <span class="n">channel</span><span class="p">:</span> <span class="o">&</span><span class="nv">'a</span> <span class="n">Channel</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">,</span> 306<span class="p">}</span> 307 308<span class="k">impl</span><span class="o"><</span><span class="nv">'a</span><span class="p">,</span> <span class="n">T</span><span class="o">></span> <span class="n">Future</span> <span class="k">for</span> <span class="n">Recv</span><span class="o"><</span><span class="nv">'a</span><span class="p">,</span> <span class="n">T</span><span class="o">></span> <span class="p">{</span> 309 <span class="k">type</span> <span class="n">Output</span> <span class="o">=</span> <span class="n">T</span><span class="p">;</span> 310 311 <span class="k">fn</span> <span class="nf">poll</span><span class="p">(</span><span class="k">self</span><span class="p">:</span> <span class="nb">Pin</span><span class="o"><&</span><span class="k">mut</span> <span class="k">Self</span><span class="o">></span><span class="p">,</span> <span class="n">cx</span><span class="p">:</span> <span class="o">&</span><span class="k">mut</span> <span class="n">Context</span><span class="o"><</span><span class="nv">'_</span><span class="o">></span><span class="p">)</span> <span class="k">-></span> <span class="n">Poll</span><span class="o"><</span><span class="k">Self</span><span class="p">::</span><span class="n">Output</span><span class="o">></span> <span class="p">{</span> 312 <span class="c1">// Is the queue empty?</span> 313 <span class="k">if</span> <span class="k">let</span> <span class="nf">Some</span><span class="p">(</span><span class="n">element</span><span class="p">)</span> <span class="o">=</span> <span class="k">self</span><span class="py">.channel.q</span><span class="nf">.pop_front</span><span class="p">()</span> <span class="p">{</span> 314 <span class="c1">// The queue has an element, so return it.</span> 315 <span class="nn">Poll</span><span class="p">::</span><span class="nf">Ready</span><span class="p">(</span><span class="n">element</span><span class="p">)</span> 316 <span class="p">}</span> <span class="k">else</span> <span class="p">{</span> 317 <span class="c1">// Queue is empty so block and try again later.</span> 318 <span class="k">self</span><span class="py">.channel.waiting_for_elements</span><span class="nf">.push</span><span class="p">(</span><span class="n">cx</span><span class="nf">.waker</span><span class="p">()</span><span class="nf">.clone</span><span class="p">());</span>
319 <span class="nn">Poll</span><span class="p">::</span><span class="n">Pending</span> 320 <span class="p">}</span> 321 <span class="p">}</span> 322<span class="p">}</span> 323</code></pre></div></div> 324 325<p><strong>Note</strong>: The above code is illustrative. In reality, the channel has 326a <code class="language-plaintext highlighter-rouge">Mutex</code> and some condition variables, but thatâs incidental for this 327post.</p> 328 329<p>If the queue is empty when <code class="language-plaintext highlighter-rouge">recv()</code> is called, the waker is stored in 330the channel and the task enters the blocked state.</p> 331 332<p>Later, when a value is added to the queue, any waiting tasks are 333woken:</p> 334 335<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">fn</span> <span class="n">send</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">(</span><span class="n">channel</span><span class="p">:</span> <span class="o">&</span><span class="k">mut</span> <span class="n">Channel</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">,</span> <span class="n">value</span><span class="p">:</span> <span class="n">T</span><span class="p">)</span> <span class="p">{</span> 336 <span class="n">channel</span><span class="py">.q</span><span class="nf">.push_back</span><span class="p">(</span><span class="n">value</span><span class="p">);</span> 337 <span class="k">let</span> <span class="n">wakers</span> <span class="o">=</span> <span class="nn">mem</span><span class="p">::</span><span class="nf">take</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="n">channel</span><span class="py">.waiting_for_elements</span><span class="p">);</span> 338 <span class="c1">// NOTE: Mutexes are released here, before waking.</span> 339 <span class="c1">// Unless we are clever with cancellation, we have to wake all futures,</span> 340 <span class="c1">// because we don't know which, if any, will attempt the next poll.</span> 341 <span class="k">for</span> <span class="n">waker</span> <span class="k">in</span> <span class="n">wakers</span> <span class="p">{</span> 342 <span class="n">waker</span><span class="nf">.wake</span><span class="p">();</span> 343 <span class="p">}</span> 344<span class="p">}</span> 345</code></pre></div></div> 346 347<p><strong>Hereâs the issue</strong>: <code class="language-plaintext highlighter-rouge">waiting_for_elements</code> is a <code class="language-plaintext highlighter-rouge">Vec<Waker></code>. The 348channel cannot know how many tasks are blocked, so we canât use a 349fixed-size array. Using a <code class="language-plaintext highlighter-rouge">Vec</code> means we allocate memory every time we 350queue a waker. And that allocation is taken and released every time we 351have to wake.</p> 352 353<p>The result is that a naive implementation will allocate and free 354repeatedly under steady-state send and recv. Thatâs a lot of memory 355allocator traffic.</p> 356 357<h2 id="can-we-use-an-intrusive-list">Can We Use an Intrusive List?</h2> 358 359<p>The optimization I want is, rather than allocating in a <code class="language-plaintext highlighter-rouge">Vec</code> every 360time a task is blocked on the queue, can we store the list of <code class="language-plaintext highlighter-rouge">Waker</code>s 361within the blocked futures themselves? We know that we only need to 362store as many <code class="language-plaintext highlighter-rouge">Waker</code>s as blocked futures, so that should work.</p> 363 364<p>It should look something like:</p> 365 366<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">pub</span> <span class="k">struct</span> <span class="n">Channel</span><span class="o"><</span><span class="n">T</span><span class="o">></span> <span class="p">{</span> 367 <span class="n">q</span><span class="p">:</span> <span class="n">VecDeque</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">,</span> 368 <span class="c1">// Intrusive doubly-linked list head.</span> 369 <span class="n">waiting_for_elements</span><span class="p">:</span> <span class="n">WakerList</span><span class="p">,</span> 370<span class="p">}</span> 371 372<span class="k">fn</span> <span class="n">
372send</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">(</span><span class="n">channel</span><span class="p">:</span> <span class="o">&</span><span class="n">Channel</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">,</span> <span class="n">value</span><span class="p">:</span> <span class="n">T</span><span class="p">)</span> <span class="p">{</span> 373 <span class="n">channel</span><span class="py">.q</span><span class="nf">.push_back</span><span class="p">(</span><span class="n">value</span><span class="p">);</span> 374 <span class="k">let</span> <span class="n">wakers</span> <span class="o">=</span> <span class="n">channel</span><span class="py">.waiting_for_elements</span><span class="nf">.extract_list</span><span class="p">();</span> 375 <span class="c1">// Release any mutex before waking.</span> 376 <span class="k">for</span> <span class="n">waker</span> <span class="k">in</span> <span class="n">wakers</span> <span class="p">{</span> 377 <span class="n">waker</span><span class="nf">.wake</span><span class="p">();</span> 378 <span class="p">}</span> 379<span class="p">}</span> 380 381<span class="k">pub</span> <span class="k">struct</span> <span class="n">Recv</span><span class="o"><</span><span class="nv">'a</span><span class="p">,</span> <span class="n">T</span><span class="o">></span> <span class="p">{</span> 382 <span class="n">channel</span><span class="p">:</span> <span class="o">&</span><span class="nv">'a</span> <span class="n">Channel</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">,</span> 383 <span class="c1">// Every Future gets a WakerSlot, which is an intrusive doubly-linked</span> 384 <span class="c1">// list node protected by Channel's mutex.</span> 385 <span class="n">waker</span><span class="p">:</span> <span class="n">WakerSlot</span><span class="p">,</span> 386<span class="p">}</span> 387 388<span class="k">impl</span><span class="o"><</span><span class="nv">'a</span><span class="p">,</span> <span class="n">T</span><span class="o">></span> <span class="n">Future</span> <span class="k">for</span> <span class="n">Recv</span><span class="o"><</span><span class="nv">'a</span><span class="p">,</span> <span class="n">T</span><span class="o">></span> <span class="p">{</span> 389 <span class="k">type</span> <span class="n">Output</span> <span class="o">=</span> <span class="n">T</span><span class="p">;</span> 390 391 <span class="k">fn</span> <span class="nf">poll</span><span class="p">(</span><span class="k">self</span><span class="p">:</span> <span class="nb">Pin</span><span class="o"><&</span><span class="k">mut</span> <span class="k">Self</span><span class="o">></span><span class="p">,</span> <span class="n">cx</span><span class="p">:</span> <span class="o">&</span><span class="k">mut</span> <span class="n">Context</span><span class="o"><</span><span class="nv">'_</span><span class="o">></span><span class="p">)</span> <span class="k">-></span> <span class="n">Poll</span><span class="o"><</span><span class="k">Self</span><span class="p">::</span><span class="n">Output</span><span class="o">></span> <span class="p">{</span> 392 <span class="k">if</span> <span class="k">let</span> <span class="nf">Some</span><span class="p">(</span><span class="n">element</span><span class="p">)</span> <span class="o">=</span> <span class="k">self</span><span class="py">.channel.q</span><span class="nf">.pop_front</span><span class="p">()</span> <span class="p">{</span> 393 <span class="nn">Poll</span><span class="p">::</span><span class="nf">Ready</span><span class="p">(</span><span class="n">element</span><span class="p">)</span> 394 <span class="p">}</span> <span class="k">else</span> <span class="p">{</span> 395 <span class="c1">// Queue is empty so try again later.</span> 396 <span class="c1">// Store the Waker in this Future by linking it into channel's list.</span> 397 <span class="k">self</span><span class="py">.channel.waiting_for_elements</span><span class="nf">.link</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="k">
397self</span><span class="py">.waker</span><span class="p">,</span> <span class="n">cx</span><span class="nf">.waker</span><span class="p">()</span><span class="nf">.clone</span><span class="p">());</span> 398 <span class="nn">Poll</span><span class="p">::</span><span class="n">Pending</span> 399 <span class="p">}</span> 400 <span class="p">}</span> 401<span class="p">}</span> 402</code></pre></div></div> 403 404<p>And thatâs about the limit of my fake illustrative code. Itâs time to 405get into details. How do we express an intrusive linked list in Rust?</p> 406 407<h2 id="intrusive-list-crates">Intrusive List Crates</h2> 408 409<p>This isnât a new idea. I looked for existing crates:</p> 410 411<ul> 412 <li><a href="https://docs.rs/intrusive-collections/">intrusive-collections</a> 413is popular, but list nodes must outlive the list itself. In my case, 414the future will never outlive the channel.</li> 415 <li><a href="https://docs.rs/futures-intrusive/">futures-intrusive</a> 416is a nice-looking crate that performs the same optimization, but 417does not meet my design goals.</li> 418 <li><a href="https://github.com/pcwalton/multilist">multilist</a> is Patrick 419Waltonâs pre-1.0 experiment. Interesting idea, but it allocates 420nodes on the heap.</li> 421</ul> 422 423<p>There are two other production examples of this approach:</p> 424 425<ul> 426 <li><a href="https://docs.rs/lilos-list/0.1.0/lilos_list/">lilos-list</a> is part 427of Cliff Biffleâs embedded operating system 428<a href="https://docs.rs/lilos/">lilos</a>. It stores wakers with an intrusive 429list. Itâs close to what I wanted, but embedded OS code tends to 430have its own concurrency model. In particular, it would take some 431work to integrate it with standard library mutexes, and it <a href="https://users.rust-lang.org/t/should-locks-be-dropped-before-calling-waker-wake/53057/4">calls 432wakers while locks are held, which is a bad 433idea</a>. 434On the other hand, since there are no threads in lilos, it can avoid 435implementing <code class="language-plaintext highlighter-rouge">Send</code> and use 436<a href="https://doc.rust-lang.org/std/cell/struct.Cell.html"><code class="language-plaintext highlighter-rouge">std::cell::Cell</code></a> 437to temporarily perform mutations on otherwise shared references.</li> 438 <li><a href="https://github.com/tokio-rs/tokio/blob/c8f3539bc11e57843745c68ee60ca5276248f9f9/tokio/src/sync/batch_semaphore.rs#L35">tokio</a> 439stores its channel wakers in an intrusive linked list too. Its 440implementation has a surprising amount of code, but itâs closest to 441what I want. The point is moot: itâs an implementation detail not 442visible outside of Tokio. (See Alice Rhylâs <a href="https://gist.github.com/Darksonn/1567538f56af1a8038ecc3c664a42462">musings on the 443challenges of intrusive structures in 444Rust</a>.)</li> 445</ul> 446 447<h2 id="pinning">Pinning</h2> 448 449<p>Itâs time to write a crate.</p> 450 451<p>I want the channel to store a <code class="language-plaintext highlighter-rouge">WakerList</code> and each future to have a 452<code class="language-plaintext highlighter-rouge">WakerSlot</code> member. Slots can be linked into the list and unlinked 453either on wake or future cancellation.</p> 454 455<p><code class="language-plaintext highlighter-rouge">WakerList</code> and <code class="language-plaintext highlighter-rouge">WakerSlot</code> form a self-referential data structure. 456Self-referential data structures are a well-known challenge in Rust. 457They require unsafe code.</p> 458 459<p>In C++, this is relatively easy. You delete the move and copy 460operations and fix up the link pointers as appropriate.</p> 461 462<p>So, at this point in my Rust journey, still thinking in C++, I assume 463âeasy!â</p> 464 465<p>I just need to disable movement with <code class="language-plaintext highlighter-rouge">!Unpin</code> (actually 466<a href="https://doc.rust-lang.org/std/marker/struct.PhantomPinned.html"><code class="language-plaintext highlighter-rouge">PhantomPinned</code></a>) 467and ensure all methods take <code class="language-plaintext highlighter-rouge">Pin<&mut WakerList></code> and <code class="language-plaintext highlighter-rouge">Pin<&mut 468WakerSlot></code>.</p> 469 470<p>Once you observe a <code class="language-plaintext highlighter-rouge">Pin<&mut T></code>, you can assume T will never move
471again. Iâm not going to discuss <code class="language-plaintext highlighter-rouge">Pin</code> in depth â Jon Gjengset has an 472<a href="https://www.youtube.com/watch?v=DkMwYxfSYNQ">excellent video describing its rationale and 473usage</a>.</p> 474 475<p>Hereâs where things start to get hard. Pinning was added well after 476Rust 1.0 was stabilized. The language pervasively assumes values of 477any type can be moved with a memcpy, so writing a data structure that 478violates that assumption makes the public APIs themselves awkward.</p> 479 480<p>Hereâs what I tried first:</p> 481 482<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">struct</span> <span class="nf">WakerSlot</span><span class="p">(</span><span class="o">...</span><span class="p">);</span> 483<span class="k">struct</span> <span class="nf">WakerList</span><span class="p">(</span><span class="o">...</span><span class="p">);</span> 484 485<span class="k">impl</span> <span class="n">WakerList</span> <span class="p">{</span> 486 <span class="k">fn</span> <span class="n">link</span><span class="o"><</span><span class="nv">'list</span> <span class="p">:</span> <span class="nv">'slot</span><span class="p">,</span> <span class="nv">'slot</span><span class="o">></span><span class="p">(</span> 487 <span class="k">self</span><span class="p">:</span> <span class="nb">Pin</span><span class="o"><&</span><span class="nv">'list</span> <span class="k">mut</span> <span class="n">WakerList</span><span class="o">></span><span class="p">,</span> 488 <span class="n">slot</span><span class="p">:</span> <span class="nb">Pin</span><span class="o"><&</span><span class="nv">'slot</span> <span class="k">mut</span> <span class="n">WakerSlot</span><span class="o">></span><span class="p">,</span> 489 <span class="p">)</span> 490<span class="p">}</span> 491</code></pre></div></div> 492 493<p>My thought was that the act of linking should constrain the 494âpinnednessâ and the lifetimes: list must outlive slot. Alas, 495<a href="https://stackoverflow.com/questions/66017394/does-rust-narrow-lifetimes-to-satisfy-constraints-defined-on-them">lifetimes donât work like 496that</a>. 497A function call cannot constrain the actual lifetimes of its parameters. 498The borrow checker will happily subset <code class="language-plaintext highlighter-rouge">'list</code> and <code class="language-plaintext highlighter-rouge">'slot</code> until it 499proves whether it can satisfy the constraints. The result is that the 500<code class="language-plaintext highlighter-rouge">link</code>âs definition above has no effect.</p> 501 502<p>The idea that lifetimes precisely match the lives of actual values is 503apparently a common misconception, and it resulted in some puzzling 504error messages.</p> 505 506<p>(Writing a post like this after the fact feels weird because âof 507course it doesnât work that wayâ but Iâm faithfully documenting my 508learning process.)</p> 509 510<p>Can <code class="language-plaintext highlighter-rouge">WakerSlot</code> itself take a lifetime to the list it references?</p> 511 512<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">struct</span> <span class="nf">WakerList</span><span class="p">(</span><span class="o">...</span><span class="p">);</span> 513<span class="k">struct</span> <span class="n">WakerSlot</span><span class="o"><</span><span class="nv">'list</span><span class="o">></span><span class="p">(</span><span class="o">...</span><span class="p">);</span> 514 515<span class="k">impl</span> <span class="n">WakerSlot</span><span class="o"><</span><span class="nv">'_</span><span class="o">></span> <span class="p">{</span> 516 <span class="k">fn</span> <span class="nf">new</span><span class="p">(</span><span class="n">list</span><span class="p">:</span> <span class="o">&</span><span class="n">WakerList</span><span class="p">)</span> <span class="k">-></span> <span class="n">WakerSlot</span><span class="o"><</span><span class="nv">'_</span><span class="o">></span><span class="p">;</span> 517<span class="p">}</span> 518</code></pre></div></div> 519 520<p>This doesnât work. If you pretend the WakerSlot has a reference to the 521WakerList, then you can never create a <code class="language-plaintext highlighter-rouge">&mut WakerList</code> elsewhere, 522because Rustâs core lifetime rule is that you can either have one mut 523reference or many shared references but never both.</p> 524 525<p>Iâm hoping this is possible and a reader leaves me a note.</p> 526 527<h2 id="when-panicking-isnt-safe-enough">When Panicking Isnât Safe Enough</h2> 528 529<p>Conceptually, <code class="language-plaintext highlighter-rouge">
529link</code> and <code class="language-plaintext highlighter-rouge">unlink</code> operations take mutable references 530to both the list and slot. But I never found a way to satisfy all of 531the rules in the type system:</p> 532 533<ul> 534 <li><code class="language-plaintext highlighter-rouge">WakerSlot</code>âs lifetime parameter does not outlive its list.</li> 535 <li>Only one <code class="language-plaintext highlighter-rouge">&mut</code> reference at a time.</li> 536 <li>Never <code class="language-plaintext highlighter-rouge">&mut</code> and <code class="language-plaintext highlighter-rouge">&</code> simultaneously.</li> 537</ul> 538 539<p>Here, I gave up on trying to express the rules in the type system and 540chose to assert at runtime. The runtime lifetime rules are:</p> 541 542<p><code class="language-plaintext highlighter-rouge">WakerList</code> must be empty when dropped. Otherwise, slots would have pointers 543to invalid memory.</p> 544 545<p><code class="language-plaintext highlighter-rouge">WakerSlot</code> must be unlinked when dropped. Otherwise, the list 546references deallocated memory.</p> 547 548<p>Reporting these invariant violations with panic is not sufficient. 549Panics can be caught, but the program would remain in a state where 550safe code can access dangling pointers and cause undefined behavior (UB).</p> 551 552<p>Therefore, when an invariant is violated, the program must abort.</p> 553 554<p>But I canât just call abort: I want this utility crate to be 555<code class="language-plaintext highlighter-rouge">[no_std]</code>, so itâs up to the calling program to decide how it aborts.</p> 556 557<p>The simplest solution I found was to panic from an <code class="language-plaintext highlighter-rouge">extern "C"</code> 558function and let Rust translate that to an abort.</p> 559 560<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">#[allow(non_snake_case)]</span> 561<span class="nd">#[inline(never)]</span> 562<span class="nd">#[cold]</span> 563<span class="k">extern</span> <span class="s">"C"</span> <span class="k">fn</span> <span class="nf">MUST_UNLINK_WakerSlot_BEFORE_DROP</span><span class="p">()</span> <span class="k">-></span> <span class="o">!</span> <span class="p">{</span> 564 <span class="c1">// panic! from extern "C" is an abort with an error message.</span> 565 <span class="nd">panic!</span><span class="p">(</span><span class="s">"Must unlink WakerSlot before drop"</span><span class="p">)</span> 566 <span class="c1">// Another option, at the cost of a tiny, stable, dependency, is</span> 567 <span class="c1">// the `abort` crate.</span> 568 <span class="c1">//abort::abort()</span> 569<span class="p">}</span> 570</code></pre></div></div> 571 572<h2 id="structural-pinning">Structural Pinning</h2> 573 574<p>I elided this detail in the example code above, but <code class="language-plaintext highlighter-rouge">WakerList</code> is 575intended to be accessed behind a mutex. However, neither 576<a href="https://doc.rust-lang.org/std/sync/struct.Mutex.html"><code class="language-plaintext highlighter-rouge">std::sync::Mutex</code></a> 577nor 578<a href="https://docs.rs/parking_lot/latest/parking_lot/type.Mutex.html"><code class="language-plaintext highlighter-rouge">parking_lot::Mutex</code></a> 579have <a href="https://doc.rust-lang.org/std/pin/index.html#projections-and-structural-pinning">structural 580pinning</a>. 581That is, <code class="language-plaintext highlighter-rouge">lock()</code> is <code class="language-plaintext highlighter-rouge">&Mutex<T></code> onto <code class="language-plaintext highlighter-rouge">&mut T</code>, allowing T to be 582moved.</p> 583 584<p>I needed a safe API for getting <code class="language-plaintext highlighter-rouge">Pin<&mut T></code> from <code class="language-plaintext highlighter-rouge">Pin<&Mutex<T>></code>.</p> 585 586<p>So I wrote the <a href="https://docs.rs/pinned-mutex/">pinned-mutex</a> crate 587which provides structurally-pinned <code class="language-plaintext highlighter-rouge">Mutex</code>, <code class="language-plaintext highlighter-rouge">MutexGuard</code>, and 588<code class="language-plaintext highlighter-rouge">Condvar</code> wrappers.</p> 589 590<p>Note that there is a <a href="https://docs.rs/pinarcmutex/">pinarcmutex crate</a> 591that offers a <code class="language-plaintext highlighter-rouge">PinArcMutex<T></code> type roughly equivalent to 592<code class="language-plaintext highlighter-rouge">Pin<Arc<Mutex<T>>></code> except with structural pinning. But it allocates 593and you canât drop in <code class="language-plaintext highlighter-rouge">parking_lot</code>âs mutex, which is faster and 594lighter than the standard libraryâs.</p> 595 596<p>
596We can imagine a future Rust version where pinning is more natural and 597has pervasive (or implicit) standard library support.</p> 598 599<h2 id="pinning-ergonomics">Pinning Ergonomics</h2> 600 601<p>Boats recently wrote a <a href="https://without.boats/blog/pin/">nice overview of why Pin is shaped the way it 602is and why it is painful to use</a>.</p> 603 604<p>And the internet is full of threads like <a href="https://www.reddit.com/r/rust/comments/v64nej/pin_suffering_continues/">âPin Suffering 605Continuesâ</a>.</p> 606 607<p>If you want to use pinned APIs safely in your own code, you will need 608to depend on a pin-projection crate like 609<a href="https://docs.rs/pin-project/">pin-project</a> or 610<a href="https://docs.rs/pin-project-lite/">pin-project-lite</a>. (And donât 611forget 612<a href="https://docs.rs/pin-project/latest/pin_project/attr.pinned_drop.html">pinned_drop</a>!)</p> 613 614<p>It works fine, but you end up with code that looks like the following.</p> 615 616<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">let</span> <span class="k">mut</span> <span class="n">wakers</span> <span class="o">=</span> <span class="n">state</span><span class="nf">.as_mut</span><span class="p">()</span><span class="nf">.project</span><span class="p">()</span><span class="py">.base</span><span class="nf">.project</span><span class="p">()</span><span class="py">.rx_wakers</span><span class="nf">.extract_some_wakers</span><span class="p">();</span> 617<span class="k">while</span> <span class="n">wakers</span><span class="nf">.wake_all</span><span class="p">()</span> <span class="p">{</span> 618 <span class="k">let</span> <span class="k">mut</span> <span class="n">state</span> <span class="o">=</span> <span class="k">self</span><span class="nf">.project_ref</span><span class="p">()</span><span class="py">.state</span><span class="nf">.lock</span><span class="p">();</span> 619 <span class="n">wakers</span><span class="nf">.extract_more</span><span class="p">(</span><span class="n">state</span><span class="nf">.as_mut</span><span class="p">()</span><span class="nf">.base</span><span class="p">()</span><span class="nf">.project</span><span class="p">()</span><span class="py">.rx_wakers</span><span class="p">);</span> 620<span class="p">}</span> 621</code></pre></div></div> 622 623<p>Writing code this way is miserable. The compiler will guide you, but 624my mind was shouting âyou know what I want, just do itâ the whole 625time.</p> 626 627<h2 id="it-works">It Works!</h2> 628 629<p>Deep breath. Tests passed. <code class="language-plaintext highlighter-rouge">WakerList</code> and <code class="language-plaintext highlighter-rouge">WakerSlot</code> provide the 630interface I wanted, so I published them in a 631<a href="https://docs.rs/wakerset/">wakerset</a> crate. It offers a safe, 632intrusive, <code class="language-plaintext highlighter-rouge">no_std</code> list of Wakers.</p> 633 634<p>With it, I could remove the main source of steady-state allocations in 635<code class="language-plaintext highlighter-rouge">batch_channel</code>.</p> 636 637<p>It was time to polish it up and ensure I didnât miss anything, and 638this blog post is only half-done.</p> 639 640<h2 id="undefined-behavior-sanitizers-and-miri">Undefined Behavior, Sanitizers, and MIRI</h2> 641 642<p>One of my original goals for <code class="language-plaintext highlighter-rouge">batch-channel</code> was to avoid unsafe 643code.</p> 644 645<p>Unfortunately, two optimizations required it. Besides the intrusive 646list described above, the MPMC channel objects themselves are managed 647with two reference counts, one for senders and one for receivers. A 648split reference count is required: when one half of the channel is 649dropped, the channel is closed.</p> 650 651<p>To simplify auditing, I placed all of this new unsafe code behind safe 652APIs and separate crates. They are:</p> 653<ul> 654 <li><a href="https://docs.rs/splitrc/">splitrc</a></li> 655 <li><a href="https://docs.rs/wakerset/">wakerset</a></li> 656 <li><a href="https://docs.rs/pinned-mutex/">pinned-mutex</a></li> 657</ul> 658 659<p>Safe Rust, excepting compiler bugs and incorrectly-designed unsound 660APIs, has no undefined behavior. Unsafe Rust, on the other hand, 661removes the guardrails and opens a <a href="https://doc.rust-lang.org/reference/behavior-considered-undefined.html">buffet of possible 662UB</a>.</p> 663 664<p>There are three ways you can deal with potential undefined behavior. 665In increasing order of happiness over time:</p> 666 667<ul> 668 <li>Hope for the best and deal with potential bugs when they come up.</li> 669 <li>Think carefully and convince yourself the code is correct.</li> 670 <li>Automated sanitizers.</li> 671</ul> 672 673<p>Fortunately, Rust supports 674<a href="https://doc.rust-lang.org/nightly/unstable-book/compiler-flags/sanitizer.html">sanitizers</a> 675that detect various types of undefined behavior. The two most useful 676are 677<a href="https://doc.rust-lang.org/nightly/unstable-book/compiler-flags/sanitizer.html#addresssanitizer">ASan</a> 678and 679<a href="https://doc.rust-lang.org/nightly/unstable-book/compiler-flags/sanitizer.html#threadsanitizer">TSan</a>. 680C++ programmers are quite familiar with them at this point, and I 681consider them table stakes for any C or C++ project.</p> 682 683<p>But Rust has one even better: 684<a href="https://github.com/rust-lang/miri">MIRI</a>
684. It catches violations of 685the Rust aliasing model, which is pickier than ASAN. Satisfying MIRI 686is where I spent most of my time.</p> 687 688<h2 id="rust-aliasing-model">Rust Aliasing Model</h2> 689 690<p>The first time I ran MIRI, it failed:</p> 691 692<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>test link_and_notify_all ... 693error: Undefined Behavior: trying to retag from <254318> for SharedReadWrite permission at alloc88289[0x10], but that tag does not exist in the borrow stack for this location 694... 695trying to retag from <254318> for SharedReadWrite permission at alloc88289[0x10], but that tag does not exist in the borrow stack for this location 696... 697help: this indicates a potential bug in the program: it performed an invalid operation, but the Stacked Borrows rules it violated are still experimental 698help: see https://github.com/rust-lang/unsafe-code-guidelines/blob/master/wip/stacked-borrows.md for further information 699</code></pre></div></div> 700 701<p>Stacked borrows? Whatâs all this?</p> 702 703<p>Here, I realized my first mistake: I dove straight into unsafe Rust 704and should have read more in advance:</p> 705<ul> 706 <li><a href="https://rust-unofficial.github.io/too-many-lists/index.html">Learning Rust With Entirely Too Many Linked 707Lists</a></li> 708 <li><a href="https://rust-lang.github.io/unsafe-code-guidelines/">Rust Unsafe Code 709Guidelines</a></li> 710 <li><a href="https://doc.rust-lang.org/nomicon/">The Rustonomicon</a></li> 711</ul> 712 713<p>My other mistake was âthinking in Câ. Being deeply familiar with <a href="https://en.wikipedia.org/wiki/Alias_analysis#Type-based_alias_analysis">Câs 714semantics</a> 715wasnât helpful here. In hindsight, it feels foolish to have assumed 716Rustâs aliasing model was similar to Câs. In C, usually, having 717pointers to values carries no meaning. Primarily you reason about 718pointer dereferencing operations.</p> 719 720<p>For example, in C, this is perfectly legal if <code class="language-plaintext highlighter-rouge">p == q</code>:</p> 721 722<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">void</span> <span class="nf">foo</span><span class="p">(</span><span class="kt">int</span><span class="o">*</span> <span class="n">p</span><span class="p">,</span> <span class="k">const</span> <span class="kt">int</span><span class="o">*</span> <span class="n">q</span><span class="p">)</span> <span class="p">{</span> 723 <span class="n">printf</span><span class="p">(</span><span class="s">"%d</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="o">*</span><span class="n">q</span><span class="p">);</span> 724 <span class="o">*</span><span class="n">p</span> <span class="o">=</span> <span class="mi">456</span><span class="p">;</span> 725 <span class="n">printf</span><span class="p">(</span><span class="s">"%d</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="o">*</span><span class="n">q</span><span class="p">);</span> <span class="c1">// if p == q, prints 456</span> 726 <span class="o">*</span><span class="p">(</span><span class="kt">int</span><span class="o">*</span><span class="p">)</span><span class="n">q</span> <span class="o">=</span> <span class="mi">789</span><span class="p">;</span> 727 <span class="n">printf</span><span class="p">(</span><span class="s">"%d</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="o">*</span><span class="n">p</span><span class="p">);</span> <span class="c1">// if p == q, prints 789</span> 728<span class="p">}</span> 729</code></pre></div></div> 730 731<p><code class="language-plaintext highlighter-rouge">const</code> is âmeaninglessâ in that it does not prevent the pointee from 732changing, so the compiler canât optimize based on it.</p> 733 734<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">fn</span> <span class="nf">foo</span><span class="p">(</span><span class="n">a</span><span class="p">:</span> <span class="o">&</span><span class="nb">u32</span><span class="p">,</span> <span class="n">b</span><span class="p">:</span> <span class="o">&</span><span class="k">mut</span> <span class="nb">u32</span><span class="p">)</span> <span class="p">{</span> 735 <span class="nd">println!</span><span class="p">(</span><span class="s">"{a}"</span><span class="p">);</span> 736 <span class="o">*</span><span class="n">b</span> <span class="o">=</span> <span class="mi">123</span><span class="p">;</span> 737 <span class="nd">println!</span><span class="p">(</span><span class="s">"{a}"</span><span class="p">);</span> <span class="c1">// always prints the same value as above</span> 738<span class="p">}</span> 739</code></pre></div></div> 740 741<p>On the other hand, Rustâs primary aliasing rule is that, at any point, 742an object may have a unique <code class="language-plaintext highlighter-rouge">&mut</code> reference to it or any number of 743shared <code class="language-plaintext highlighter-rouge">&</code> references, but never both.</p> 744 745<p>The optimizer will take advantage of that. <code class="language-plaintext highlighter-rouge">a</code> is not reloaded from 746memory because the write to <code class="language-plaintext highlighter-rouge">b</code> cannot alias it.</p> 747 748<p>The <a href="https://doc.rust-lang.org/nomicon/aliasing.html">aliasing rules</a> 749in Rust are not fully defined. Thatâs part of what makes this hard. 750You have to write code assuming the most pessimal aliasing model.</p> 751 752<p>Under the most pessimal aliasing rules, you have to assume taking a 753<code class="language-plaintext highlighter-rouge">&mut</code> reference to a value <em>immediately</em> writes to it and continues 754to write to it as long as the reference lives. And if you have a 755shared reference, you have to assume the value is read at arbitrary 756times as long as any reference is held.</p> 757 758<h2 id="boxleak">Box::leak</h2> 759 760<p>Letâs start with the first confusing example I ran into:</p> 761 762<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">let</span> <span class="n">p</span> <span class="o">=</span> <span class="nn">Box</span><span class="p">::</span><span class="nf">leak</span><span class="p">(</span><span class="nn">Box</span><span class="p">::</span><span class="nf">new</span><span class="p">(</span><span class="nn">MyThing</span><span class="p">::</span><span class="nf">new</span><span class="p">()))</span> <span class="k">as</span> <span class="o">*</span><span class="k">mut</span> <span class="n">MyThing</span><span class="p">;</span> 763<span class="c1">// later:</span> 764<span class="k">let</span> <span class="k">ref</span><span class="p">:</span> <span class="o">&</span><span class="n">MyThing</span> <span class="o">=</span> <span class="o">*</span><span class="n">p</span><span class="p">;</span> 765<span class="k">ref</span><span class="nf">.method</span><span class="p">();</span> 766</code></pre></div></div> 767 768<p>MIRI failed, complaining that <code class="language-plaintext highlighter-rouge">p</code> was formed from the perpetual <code class="language-plaintext highlighter-rouge">&mut</code> 769returned by <code class="language-plaintext highlighter-rouge">Box::leak</code> and therefore itâs UB to create a shared 770<code class="language-plaintext highlighter-rouge">&MyThing</code> reference to it at any point thenceforth.</p> 771 772<p>The fix was to allocate without forming a reference:</p> 773 774<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">let</span> <span class="n">p</span> <span class="o">=</span> <span class="nn">Box</span><span class="p">::</span><span class="nf">into_raw</span><span class="p">(</span><span class="nn">MyThing</span><span class="p">::</span><span class="nf">new</span><span class="p">());</span> 775<span class="c1">// later:</span> 776<span class="k">let</span> <span class="k">ref</span><span class="p">:</span> <span class="o">&</span><span class="n">MyThing</span> <span class="o">=</span> <span class="o">*</span><span class="n">p</span><span class="p">;</span> 777<span class="k">ref</span><span class="nf">.method</span><span class="p">();</span> 778</code></pre></div></div> 779 780<p><strong>Note</strong>: This may have been a MIRI bug or the rules have since been 781relaxed, because I can no longer reproduce as of nightly-2024-06-12. 782Hereâs where the memory model and aliasing rules not being defined 783caused some pain: when MIRI fails, itâs unclear whether itâs my fault 784or not. For example, given the <code class="language-plaintext highlighter-rouge">&mut</code> was immediately turned into a 785pointer, does the <code class="language-plaintext highlighter-rouge">&mut</code> reference still exist? There are multiple 786valid interpretations of the rules.</p> 787 788<h2 id="boxfrom_raw">Box::from_raw</h2> 789 790<p>OK, if you allocate some memory with <code class="language-plaintext highlighter-rouge">Box::into_raw</code>, youâd expect to 791deallocate with <code class="language-plaintext highlighter-rouge">Box::from_raw</code>, right? That failed MIRI too.</p> 792 793<p>I ended up having to write:</p> 794 795<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">unsafe</span> <span class="p">{</span> 796 <span class="k">let</span> <span class="n">ptr</span> <span class="o">=</span> <span class="n">ptr</span><span class="nf">.as_ptr</span><span class="p">();</span> 797 <span class="nn">std</span><span class="p">::</span><span class="nn">ptr</span><span class="p">::</span><span class="nf">drop_in_place</span><span class="p">(</span><span class="n">ptr</span><span class="p">);</span> 798 <span class="nn">std</span><span class="p">::</span><span class="nn">alloc</span><span class="p">::</span><span class="nf">dealloc</span><span class="p">(</span>
799 <span class="n">ptr</span> <span class="k">as</span> <span class="o">*</span><span class="k">mut</span> <span class="nb">u8</span><span class="p">,</span> 800 <span class="nn">std</span><span class="p">::</span><span class="nn">alloc</span><span class="p">::</span><span class="nn">Layout</span><span class="p">::</span><span class="nn">new</span><span class="p">::</span><span class="o"><</span><span class="n">Inner</span><span class="o"><</span><span class="n">T</span><span class="o">>></span><span class="p">());</span> 801<span class="p">}</span> 802</code></pre></div></div> 803 804<p><strong>Note</strong>: This may have also been a MIRI bug. It is no longer 805reproducible. I changed <code class="language-plaintext highlighter-rouge">splitrc</code> to use <code class="language-plaintext highlighter-rouge">Box::into_raw</code> and 806<code class="language-plaintext highlighter-rouge">Box::from_raw</code> and it passes MIRI. I enabled MIRI in my CI so weâll 807see if it breaks again going forward.</p> 808 809<h2 id="linkage-references-and-interior-mutability">Linkage, References, and Interior Mutability</h2> 810 811<p>Thatâs channel allocation and deallocation covered. Now letâs look at 812the intrusive pointers in <code class="language-plaintext highlighter-rouge">wakerset</code>.</p> 813 814<p>In a linked list, every node has a linkage struct.</p> 815 816<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">struct</span> <span class="n">Pointers</span> <span class="p">{</span> 817 <span class="n">next</span><span class="p">:</span> <span class="o">*</span><span class="k">mut</span> <span class="n">Pointers</span><span class="p">,</span> 818 <span class="n">prev</span><span class="p">:</span> <span class="o">*</span><span class="k">mut</span> <span class="n">Pointers</span><span class="p">,</span> 819 <span class="n">pinned</span><span class="p">:</span> <span class="n">PhantomPinned</span><span class="p">,</span> 820<span class="p">}</span> 821 822<span class="k">struct</span> <span class="n">WakerList</span> <span class="p">{</span> 823 <span class="n">pointers</span><span class="p">:</span> <span class="n">Pointers</span><span class="p">,</span> 824<span class="p">}</span> 825 826<span class="k">struct</span> <span class="n">WakerSlot</span> <span class="p">{</span> 827 <span class="n">pointers</span><span class="p">:</span> <span class="n">Pointers</span><span class="p">,</span> 828 <span class="c1">// Required to assert that this slot is never</span> 829 <span class="c1">// unlinked from an unrelated list.</span> 830 <span class="n">owner</span><span class="p">:</span> <span class="o">*</span><span class="k">mut</span> <span class="n">WakerList</span><span class="p">,</span> 831 <span class="n">waker</span><span class="p">:</span> <span class="nb">Option</span><span class="o"><</span><span class="n">Waker</span><span class="o">></span><span class="p">,</span> 832<span class="p">}</span> 833</code></pre></div></div> 834 835<p>Now imagine two threads. Thread A holds a <code class="language-plaintext highlighter-rouge">&mut WakerList</code> with the 836intent to extract pending Wakers. Thread B happens to hold a 837<code class="language-plaintext highlighter-rouge">&WakerSlot</code> at the same time.</p> 838 839<p>It is UB for code traversing the pointers to form a <code class="language-plaintext highlighter-rouge">&mut WakerSlot</code> 840(or even a <code class="language-plaintext highlighter-rouge">&WakerSlot</code>) if any thread might have a <code class="language-plaintext highlighter-rouge">&mut WakerSlot</code>, 841because this violates Rustâs aliasing rules. A <code class="language-plaintext highlighter-rouge">&mut</code> reference must 842always be exclusive, <em>even if it is never dereferenced</em>. This is the 843important difference with C.</p> 844 845<p>Because Rust reorders reads and writes based on its aliasing rules, 846you must never convert a pointer into a reference unless you know that 847nobody else has a conflicting reference.</p> 848 849<p>We need to prevent the compiler from optimizing a <code class="language-plaintext highlighter-rouge">&WakerSlot</code> into 850early reads of the <code class="language-plaintext highlighter-rouge">pointers</code> and <code class="language-plaintext highlighter-rouge">waker</code> fields.</p> 851 852<p><a href="https://doc.rust-lang.org/std/cell/struct.UnsafeCell.html"><code class="language-plaintext highlighter-rouge">UnsafeCell</code></a> 853is the tool to reach for. It introduces a âmutability barrierâ, and 854<code class="language-plaintext highlighter-rouge">UnsafeCell<Pointers></code> tells Rust not to cache reads. We are 855responsible for ensuring we wonât violate Rustâs aliasing rules when 856accessing <code class="language-plaintext highlighter-rouge">Pointers</code>âs fields.</p> 857 858<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">struct</span> <span class="n">WakerList</span> <span class="p">{</span> 859 <span class="n">pointers</span><span class="p">:</span> <span class="n">Pointers</span><span class="p">,</span> 860<span class="p">}</span> 861 862<span class="k">struct</span> <span class="n">WakerSlot</span> <span class="p">{</span> 863 <span class="c1">// UnsafeCell: written by WakerList independent of</span> 864 <span class="c1">// WakerSlot references</span> 865 <span class="n">pointers</span><span class="p">:</span> <span class="n">UnsafeCell</span><span class="o"><</span><span class="n">Pointers</span><span class="o">></span><span class="p">,</span> 866 <span class="c1">// Required to assert that this slot is never</span> 867 <span class="c1">// unlinked from an unrelated list.</span> 868 <span class="n">owner</span><span class="p">:</span> <span class="o">*</span><span class="k">mut</span> <span class="n">WakerList</span><span class="p">,</span> 869 <span class="n">waker</span><span class="p">:</span> <span class="nb">Option</span><span class="o"><</span><span class="n">
869Waker</span><span class="o">></span><span class="p">,</span> 870<span class="p">}</span> 871</code></pre></div></div> 872 873<p>A circular linked list means that only a slot reference is required to 874mutate the list, so I needed to enforce the guarantee that only one 875thread may access the <code class="language-plaintext highlighter-rouge">UnsafeCell</code>s at a time.</p> 876 877<p>I did this with an important, if subtle, API guarantee: all link and 878unlink operations take <code class="language-plaintext highlighter-rouge">&mut WakerList</code>. If <code class="language-plaintext highlighter-rouge">&mut WakerSlot</code> was 879sufficient to unlink, it could violate thread safety if <code class="language-plaintext highlighter-rouge">WakerList</code> 880was behind a mutex. (This also means that <code class="language-plaintext highlighter-rouge">WakerList</code> does not require 881an <code class="language-plaintext highlighter-rouge">UnsafeCell<Pointers></code>.)</p> 882 883<p>The 884<a href="https://docs.rs/pinned-aliasable/latest/pinned_aliasable/"><code class="language-plaintext highlighter-rouge">pinned-aliasable</code></a> 885crate solves a related problem: how do we define self-referential data 886structures with mutable references that do not miscompile? Read the 887motivation in the crateâs doc comments. Itâs a situation required by 888async futures, which are self-referential and thus pinned, but have no 889desugaring. See the open Rust <a href="https://github.com/rust-lang/rust/issues/63818">Issue 890#63818</a>.</p> 891 892<h2 id="avoiding-references-entirely">Avoiding References Entirely</h2> 893 894<p>As mentioned, when traversing a linked list, itâs easy to form conflicting 895<code class="language-plaintext highlighter-rouge">&mut</code> references to nodes. Consider this slightly contrived unlink 896example:</p> 897 898<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">let</span> <span class="n">p</span> <span class="o">=</span> <span class="o">&</span><span class="n">slot</span><span class="py">.pointers</span> <span class="k">as</span> <span class="o">*</span><span class="k">mut</span> <span class="n">Pointers</span><span class="p">;</span> 899<span class="k">let</span> <span class="n">next</span><span class="p">:</span> <span class="o">&</span><span class="k">mut</span> <span class="n">Pointers</span> <span class="o">=</span> <span class="o">*</span><span class="p">(</span><span class="o">*</span><span class="n">p</span><span class="p">)</span><span class="py">.next</span><span class="p">;</span> 900<span class="k">let</span> <span class="n">prev</span><span class="p">:</span> <span class="o">&</span><span class="k">mut</span> <span class="n">Pointers</span> <span class="o">=</span> <span class="o">*</span><span class="p">(</span><span class="o">*</span><span class="n">p</span><span class="p">)</span><span class="py">.prev</span><span class="p">;</span> 901<span class="n">next</span><span class="py">.prev</span> <span class="o">=</span> <span class="n">prev</span> <span class="k">as</span> <span class="o">*</span><span class="k">mut</span> <span class="n">Pointers</span><span class="p">;</span> 902<span class="n">prev</span><span class="py">.next</span> <span class="o">=</span> <span class="n">next</span> <span class="k">as</span> <span class="o">*</span><span class="k">mut</span> <span class="n">Pointers</span><span class="p">;</span> 903<span class="p">(</span><span class="o">*</span><span class="n">p</span><span class="p">)</span><span class="py">.next</span> <span class="o">=</span> <span class="nn">ptr</span><span class="p">::</span><span class="nf">null_mut</span><span class="p">();</span> 904<span class="p">(</span><span class="o">*</span><span class="n">p</span><span class="p">)</span><span class="py">.prev</span> <span class="o">=</span> <span class="nn">ptr</span><span class="p">::</span><span class="nf">null_mut</span><span class="p">();</span> 905</code></pre></div></div> 906 907<p>If <code class="language-plaintext highlighter-rouge">slot</code> is the only slot in the list, then <code class="language-plaintext highlighter-rouge">next</code> and <code class="language-plaintext highlighter-rouge">prev</code> both 908point to the <code class="language-plaintext highlighter-rouge">WakerList</code> and weâve now formed two mut references to 909the same value, which is UB.</p> 910 911<p>In this particular case, we could ensure every pointer dereference 912occurs as a temporary. That limits the scope of each reference to each 913line.</p> 914 915<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">let</span> <span class="n">p</span> <span class="o">=</span> <span class="o">&</span><span class="n">slot</span><span class="py">.pointers</span> <span class="k">as</span> <span class="o">*</span><span class="k">mut</span> <span class="n">Pointers</span><span class="p">;</span> 916<span class="k">let</span> <span class="n">next</span> <span class="o">=</span> <span class="p">(</span><span class="o">*</span><span class="n">p</span><span class="p">)</span><span class="py">.next</span><span class="p">;</span> 917<span class="k">let</span> <span class="n">prev</span> <span class="o">=</span> <span class="p">(</span><span class="o">*</span><span class="n">p</span><span class="p">)</span><span class="py">.prev</span><span class="p">;</span> 918<span class="p">(</span><span class="o">*</span><span class="n">next</span><span class="p">)</span><span class="py">.prev</span> <span class="o">=</span> <span class="n">prev</span><span class="p">;</span> 919<span class="p">(</span><span class="o">*</span><span class="n">prev</span><span class="p">)</span><span class="py">.next</span> <span class="o">=</span> <span class="n">next</span><span class="p">;</span> 920<span class="p">(</span><span class="o">*</span><span class="n">p</span><span class="p">)</span><span class="py">.next</span> <span class="o">=</span> <span class="nn">ptr</span><span class="p">::</span><span class="nf">null_mut</span><span class="p">();</span> 921<span class="p">(</span><span class="o">*</span><span class="n">p</span><span class="p">)</span><span class="py">.prev</span> <span class="o">=</span> <span class="nn">ptr</span><span class="p">::</span><span class="nf">null_mut</span><span class="p">();</span> 922</code></pre></div></div> 923 924<p>But I just donât trust myself to ensure, under all code paths, that I 925never have two references overlap, violating Rustâs aliasing rules.</p> 926 927<p>Itâs kind of miserable, but the safest approach is to avoid creating 928references entirely and operate entirely in the domain of pointer 929reads, writes, and offsets.</p> 930 931<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">let</span> <span class="n">nextp</span> <span class="o">=</span> <span class="nd">addr_of_mut!</span><span class="p">((</span><span class="o">*</span><span class="n">node</span><span class="p">)</span><span class="py">.next</span><span class="p">);</span> 932<span class="k">let</span> <span class="n">prevp</span> <span class="o">=</span> <span class="nd">addr_of_mut!</span><span class="p">((</span><span class="o">*</span><span class="n">node</span><span class="p">)</span><span class="py">.prev</span><span class="p">);</span> 933<span class="k">let</span> <span class="n">next</span> <span class="o">=</span> <span class="n">nextp</span><span class="nf">.read</span><span class="p">();</span> 934<span class="k">let</span> <span class="n">prev</span> <span class="o">=</span> <span class="n">prevp</span><span class="nf">.read</span><span class="p">();</span> 935<span class="nd">addr_of_mut!</span><span class="p">((</span><span class="o">*</span><span class="n">prev</span><span class="p">)</span><span class="py">.next</span><span class="p">)</span><span class="nf">.write</span><span class="p">(</span><span class="n">next</span><span class="p">);</span> 936<span class="nd">addr_of_mut!</span><span class="p">((</span><span class="o">*</span><span class="n">next</span><span class="p">)</span><span class="py">.prev</span><span class="p">)</span><span class="nf">.write</span><span class="p">(</span><span class="n">prev</span><span class="p">);</span> 937<span class="n">nextp</span><span class="nf">.write</span><span class="p">(</span><span class="nn">ptr</span><span class="p">::</span><span class="nf">null_mut</span><span class="p">());</span> 938<span class="n">prevp</span><span class="nf">.write</span><span class="p">(</span><span class="nn">ptr</span><span class="p">::</span><span class="nf">null_mut</span><span class="p">());</span> 939</code></pre></div></div> 940 941<p>(So much syntax. Makes you appreciate C.)</p> 942 943<p><code class="language-plaintext highlighter-rouge">addr_of_mut!</code> is key: it computes a pointer to a place expression 944without forming a reference. There are gotchas: you can still 945accidentally form a reference within an <code class="language-plaintext highlighter-rouge">addr_of_mut!</code> argument. Read 946<a href="https://doc.rust-lang.org/beta/std/ptr/macro.addr_of_mut.html">the 947documentation</a>.</p> 948 949<p><strong>Note</strong>: As I publish this, Rust 1.82 introduces <a href="https://doc.rust-lang.org/stable/reference/expressions/operator-expr.html#raw-borrow-operators">new 950syntax</a> 951that allows replacing <code class="language-plaintext highlighter-rouge">addr_of_mut!</code> with <code class="language-plaintext highlighter-rouge">&raw mut</code> and <code class="language-plaintext highlighter-rouge">addr_of!</code> 952with <code class="language-plaintext highlighter-rouge">&raw const</code>. Itâs not yet clear to me how much this prevents 953accidental reference creation.</p> 954 955<p>
955Despite the noise, to be safe, I ended up converting all of <code class="language-plaintext highlighter-rouge">WakerSet</code> 956to pointer reads and writes. Itâs not greppable: the code looks like 957itâs still full of pointer dereferences, but theyâre within 958<code class="language-plaintext highlighter-rouge">addr_of_mut!</code> and the place expressions have the right shape.</p> 959 960<p>I think it was Patrick Walton who once proposed a sugar for unsafe 961Rust and pointers with a hypothetical <code class="language-plaintext highlighter-rouge">-></code> operator. It would be 962convenient and easier on the eyes.</p> 963 964<p>Until the Rust memory model stabilizes further and the aliasing rules 965are well-defined, your best option is to integrate ASAN, TSAN, and 966MIRI (both <a href="https://github.com/rust-lang/unsafe-code-guidelines/blob/master/wip/stacked-borrows.md">stacked 967borrows</a> 968and <a href="https://perso.crans.org/vanille/treebor/">tree borrows</a>) into 969your continuous integration for any project that contains unsafe code.</p> 970 971<p>If your project is safe Rust but depends on a crate which makes heavy 972use of unsafe code, you should probably still enable sanitizers. I 973didnât discover all UB in wakerset until it was integrated into 974batch-channel.</p> 975 976<h2 id="miri-stacked-borrows-and-tree-borrows">MIRI: Stacked Borrows and Tree Borrows</h2> 977 978<p>MIRI supports two aliasing models: <a href="https://github.com/rust-lang/unsafe-code-guidelines/blob/master/wip/stacked-borrows.md">stacked 979borrows</a> 980and <a href="https://perso.crans.org/vanille/treebor/">tree borrows</a>.</p> 981 982<p>I wonât attempt to describe them. They are different approaches with 983the same goal: formalize and validate the Rust memory model. Ralf 984Jung, Neven Villani, and all are doing amazing work. Without MIRI, it 985would be hard to trust unsafe Rust.</p> 986 987<p>I decided to run both stacked and tree borrows and havenât hit any 988false positives so far.</p> 989 990<h2 id="active-research-in-self-referential-structures">Active Research in Self-Referential Structures</h2> 991 992<p>This topic is an <a href="https://github.com/rust-lang/unsafe-code-guidelines/issues/495">active work in 993progress</a>. 994I hope this blog post is obsolete in two years.</p> 995 996<p>Self-referential and pinned data structures are something of a hot 997topic right now. The <a href="https://rust-for-linux.com/">Rust-for-Linux 998project</a>, doing the kinds of things 999systems programs do, has <code class="language-plaintext highlighter-rouge">Pin</code> ergonomics and self-referential data 1000structures near the top of <a href="https://github.com/Rust-for-Linux/linux/issues/354">their 1001wishlist</a>.</p> 1002 1003<p>In particular, <a href="https://rust-for-linux.com/the-safe-pinned-initialization-problem">pinned initialization 1004problem</a>. 1005The Linux kernel has self-referential data structures and it is 1006currently hard to initialize them with safe code. In <code class="language-plaintext highlighter-rouge">wakerset</code>, I 1007sidestepped this problem at the cost of a small amount of runtime 1008inefficiency by giving <code class="language-plaintext highlighter-rouge">WakerList</code> two empty states: one that is 1009moveable and one that is pinned. The former converts to the latter the 1010first time it is used after pinning.</p> 1011 1012<p>y86-dev has a great blog post proposing <a href="https://y86-dev.github.io/blog/safe-pinned-initialization/overview.html">Safe Pinned 1013Initialization</a>.</p> 1014 1015<h2 id="conclusions">Conclusions</h2> 1016 1017<p>As much as this post might come across as a gripefest, I still think 1018Rust is great. In particular, its composable safety. The result of my 1019pain is a safe, efficient API. You can use <code class="language-plaintext highlighter-rouge">wakerset</code> without any risk 1020of undefined behavior.</p> 1021 1022<p>What I learned from this experience:</p> 1023<ul> 1024 <li>Be extra careful with any use of <code class="language-plaintext highlighter-rouge">unsafe</code>.</li> 1025 <li>References, even if never used, are more dangerous than pointers in 1026C.</li> 1027 <li>Pinning syntax is awful, but it feels like Rust could solve this 1028someday.</li> 1029 <li><code class="language-plaintext highlighter-rouge">UnsafeCell</code> is required for intrusive structures.</li> 1030 <li>I donât know how to statically constrain lifetime relationships with 1031intrusive structures, but maybe itâs possible? Avoiding the need for 1032runtime assertions would be nice.</li> 1033 <li>MIRI, especially under multithreaded stress tests, is critical.</li> 1034 <li>Putting this in words was as hard as writing the code.</li> 1035</ul> 1036 1037 </li><li><span class="post-meta">Feb 18, 2024</span> 1038 <h3> 1039 <a class="post-link" href="/2024/02/windows-terminal-latency/"> 1040 Terminal Latency on Windows 1041 </a> 1042 </h3> 1043 <p><strong>UPDATE 2024-04-15</strong>: Windows Terminal 1.19 contains a fix that 1044reduces latency by half! Itâs now competitive with WSLtty on my 1045machine. Details in the <a href="https://github.com/microsoft/terminal/issues/5590">GitHub 1046Issue</a>.</p> 1047 1048<p>In 2009, I wrote about <a href="https://chadaustin.me/2009/10/reasons-why-mintty-is-the-best-terminal-on-windows/">why MinTTY is the best terminal on 1049Windows</a>. 1050Even today, that post is one of my most popular.</p> 1051 1052<figure> 1053<a href="/wp-uploads/mintty_right_click.png"><img src="/wp-uploads/mintty_right_click.png" alt="MinTTY in 2009" /></a> 1054<figcaption>
1054MinTTY in 2009</figcaption> 1055</figure> 1056 1057<p>Since then, the terminal situation on Windows has improved:</p> 1058<ul> 1059 <li>Cygwin defaults to MinTTY; you no longer need to manually install 1060it.</li> 1061 <li>Windows added <a href="https://devblogs.microsoft.com/commandline/windows-command-line-introducing-the-windows-pseudo-console-conpty/">PTY 1062support</a>, 1063obviating the need for offscreen console window hacks that add 1064latency.</li> 1065 <li>Windows added basically full support for <a href="https://learn.microsoft.com/en-us/windows/console/console-virtual-terminal-sequences">ANSI terminal 1066sequences</a> 1067in both the legacy conhost.exe consoles and its new <a href="https://github.com/microsoft/terminal">Windows 1068Terminal</a>.</li> 1069 <li>We now have a variety of terminals to choose from, even on Windows: 1070<a href="https://cmder.app/">Cmder</a>, <a href="https://conemu.github.io/">ConEmu</a>, 1071<a href="https://alacritty.org/">Alacritty</a>, 1072<a href="https://wezfurlong.org/wezterm/index.html">WezTerm</a>, 1073<a href="http://xtermjs.org/">xterm.js</a> (component of Visual Studio Code)</li> 1074</ul> 1075 1076<p>The beginning of a year is a great time to look at your tools and 1077improve your environment.</p> 1078 1079<p>Iâd already <a href="https://chadaustin.me/2024/01/truecolor-terminal-emacs/">enabled 24-bit color in all of my 1080environments</a> 1081and <a href="https://chadaustin.me/2024/02/tmux-config/">streamlined my tmux 1082config</a>. Itâs about time 1083that I take a look at the newer terminals.</p> 1084 1085<p>Roughly in order, I care about:</p> 1086<ul> 1087 <li>Minimum feature set: 24-bit color, reasonable default fonts with 1088emoji support, italics are nice.</li> 1089 <li>Input latency.</li> 1090 <li>Throughput at line rate, for example, when I <code class="language-plaintext highlighter-rouge">cat</code> a large file.</li> 1091 <li>Support for multiple tabs in one window would be nice, but tmux 1092suffices for me.</li> 1093</ul> 1094 1095<h2 id="which-terminals-should-i-test">Which terminals should I test?</h2> 1096 1097<p>I considered the following.</p> 1098 1099<ul> 1100 <li>Legacy conhost.exe (also known as Windows Console), Windows 10 19045</li> 1101 <li>MinTTY (3.7.0)</li> 1102 <li>Alacritty (0.13.1)</li> 1103 <li>WezTerm (20240203-110809-5046fc22)</li> 1104 <li>Windows Terminal (1.18.10301.0)</li> 1105</ul> 1106 1107<h2 id="testing-features">Testing Features</h2> 1108 1109<p>Testing color and italics support is easy with my 1110<a href="https://gist.github.com/chadaustin/2d2c2cb4b71fd1d4163aa8115077624a">colortest.rs</a> 1111script. To test basic emoji, you can cat the <a href="https://unicode.org/Public/emoji/1.0/emoji-data.txt">Unicode emoji 1.0 1112emoji-data.txt</a>. 1113To test more advanced support, try the zero-width joiner list in the 1114<a href="https://unicode.org/Public/emoji/latest/">latest/</a> directory.</p> 1115 1116<table> 1117 <thead> 1118 <tr> 1119 <th>Terminal</th> 1120 <th>Emoji</th> 1121 <th>Font Attributes</th> 1122 </tr> 1123 </thead> 1124 <tbody> 1125 <tr> 1126 <td>conhost.exe</td> 1127 <td>No</td> 1128 <td>No italics</td> 1129 </tr> 1130 <tr> 1131 <td>MinTTY</td> 1132 <td>Black and white</td> 1133 <td>All major attributes</td> 1134 </tr> 1135 <tr> 1136 <td>Alacritty</td> 1137 <td>Black and white</td> 1138 <td>Everything but double underline</td> 1139 </tr> 1140 <tr> 1141 <td>WezTerm</td> 1142 <td><a href="https://wezfurlong.org/wezterm/config/fonts.html">Color</a></td> 1143 <td>All major attributes</td> 1144 </tr> 1145 <tr> 1146 <td>Windows Terminal</td> 1147 <td>Color</td> 1148 <td>All major attributes</td> 1149 </tr> 1150 </tbody> 1151</table> 1152 1153<p>Everything but conhost.exe meets my bar.</p> 1154 1155<p>Itâs also worth noting that conhost.exe has a terrible default 1156palette. The default yellow is a pukey green and dark blue is barely 1157visible. You can change palettes, but defaults matter.</p> 1158 1159<figure> 1160<a href="/images/windows-terminal-latency/default-palette-conhost.png"><img src="/images/windows-terminal-latency/default-palette-conhost.png" alt="Conhost.exe Default Palette" /></a> 1161<figcaption>Conhost.exe Default Palette</figcaption> 1162</figure> 1163 1164<figure> 1165<a href="/images/windows-terminal-latency/default-palette-mintty.png"><img src="/images/windows-terminal-latency/default-palette-mintty.png" alt="MinTTY Default Palette" /></a> 1166<figcaption>MinTTY Default Palette</figcaption> 1167</figure> 1168 1169<h2 id="latency">Latency</h2> 1170 1171<p>I set up two latency tests. One with an 80x50 blank window in the 1172upper left corner of the screen. The other fullscreen, editing an 1173Emacs command at the bottom of the screen.</p> 1174 1175<p>Since latencies are additive, system configuration doesnât matter as 1176much as the absolute milliseconds of latency each terminal adds, but 1177Iâll describe my entire setup and include total keypress-to-pixels 1178latency.</p> 1179 1180<ul> 1181 <li>Windows 10</li> 1182 <li>Intel i7-4771 @ 3.5 GHz</li> 1183 <li>NVIDIA GTX 1060</li> 1184 <li>Keyboard: <a href="https://1upkeyboards.com/shop/keyboard-kits/macro-pads/sweet16-macro-pad-white/">Sweet 16 Macro 1185Pad</a></li> 1186 <li>Display: <a href="https://www.lg.com/us/monitors/lg-27gp950-b-gaming-monitor">LG 118727GP950-B</a> 1188at 4K, 120 Hz, adaptive sync</li> 1189</ul> 1190 1191<h3 id="measurement-methodology">Measurement Methodology</h3> 1192 1193<p>With <a href="https://isitsnappy.com/">Is It Snappy?</a>, I measured the number 1194of frames between pressing a key and pixels changing on the screen.</p> 1195 1196<p>To minimize ambiguity about when the key was pressed, I slammed a 1197pencilâs eraser into the key, and always measured the key press as the 1198<em>second</em> frame after contact. (The first frame was usually when the 1199eraser barely touched the key. It would usually clear the activation 1200depth by the second frame.)</p> 1201 1202<p>I considered the latency to end when pixels just started to change on 1203the screen. In practice, pixels take several 240 Hz frames to 1204transition from black to white, but I consistently marked the 1205beginning of that transition.</p> 1206 1207<p>I took five measurements for each configuration and picked the median. 1208Each measurement was relatively consistent, so average would have been 1209a fine metric too. It doesnât change the results below.</p> 1210 1211<h3 id="80x50">80x50</h3> 1212 1213<p>80x50 window, upper left of screen, cleared terminal, single keypress.</p> 1214 1215<p>Confirmed window size with:</p> 1216 1217<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ echo $(tput cols)x$(tput lines) 121880x50 1219</code></pre></div></div> 1220 1221<table> 1222 <thead> 1223 <tr> 1224 <th>Terminal</th> 1225 <th>Median Latency (ms)</th> 1226 <th>240 Hz Camera Frames</th> 1227 </tr> 1228 </thead> 1229 <tbody> 1230 <tr> 1231 <td>conhost.exe WSL1</td> 1232 <td>33.3</td> 1233 <td>8</td> 1234 </tr> 1235 <tr> 1236 <td>MinTTY WSL1</td> 1237 <td>33.3</td> 1238 <td>8</td> 1239 </tr> 1240 <tr> 1241 <td>conhost.exe Cygwin</td> 1242 <td>41.3</td> 1243 <td>10</td> 1244 </tr> 1245 <tr> 1246 <td>MinTTY Cygwin</td> 1247 <td>57.9</td> 1248 <td>14</td> 1249 </tr> 1250 <tr> 1251 <td>WezTerm cmd.exe</td> 1252 <td>62.5</td> 1253 <td>15</td> 1254 </tr> 1255 <tr> 1256 <td>Alacritty WSL1</td> 1257 <td>62.5</td> 1258 <td>15</td> 1259 </tr> 1260 <tr> 1261 <td>WezTerm WSL1</td> 1262 <td>66.7</td> 1263 <td>16</td> 1264 </tr> 1265 <tr> 1266 <td>Windows Terminal WSL1</td> 1267 <td>66.7</td> 1268 <td>16</td> 1269 </tr> 1270 </tbody> 1271</table> 1272 1273<h3 id="fullscreen">Fullscreen</h3> 1274 1275<p>Maximized emacs, editing a command in the bottom row of the terminal. 1276I only tested WSL1 this time.</p> 1277 1278<table> 1279 <thead> 1280 <tr> 1281 <th>Terminal</th> 1282 <th>Median Latency (ms)</th> 1283 <th>240 Hz Camera Frames</th> 1284 </tr> 1285 </thead> 1286 <tbody> 1287 <tr> 1288 <td>conhost.exe</td> 1289 <td>45.8</td> 1290 <td>11</td> 1291 </tr> 1292 <tr> 1293 <td>MinTTY</td> 1294 <td>52.42</td> 1295 <td>12</td> 1296 </tr> 1297 <tr> 1298 <td>WezTerm</td> 1299 <td>75</td> 1300 <td>18</td> 1301 </tr> 1302 <tr> 1303 <td>Windows Terminal</td> 1304 <td>75</td> 1305 <td>18</td> 1306 </tr> 1307 <tr> 1308 <td>Alacritty</td> 1309 <td>87.5</td> 1310 <td>21</td> 1311 </tr> 1312 </tbody> 1313</table> 1314 1315<h3 id="throughput">Throughput</h3> 1316 1317<p>I generated a 100,000-line file with:</p> 1318 1319<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ yes "This sentence has forty-five (45) characters." | head -n 100000 > /tmp/lines.txt 1320</code></pre></div></div> 1321 1322<p>Then I measured the wall-clock duration of:</p> 1323 1324<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ time cat /tmp/lines.txt 1325</code></pre></div></div> 1326 1327<p>This benchmark captures the case that I accidentally dump a ton of 1328output and Iâm sitting there just waiting for the terminal to become 1329responsive again. I have a gigabit internet connection, and itâs 1330embarrassing to be CPU-bound instead of IO-bound.</p> 1331 1332<p>
1332I did include Cygwin in this test, just to have two different MinTTY 1333datapoints.</p> 1334 1335<table> 1336 <thead> 1337 <tr> 1338 <th>Terminal</th> 1339 <th>Elapsed Time (s)</th> 1340 </tr> 1341 </thead> 1342 <tbody> 1343 <tr> 1344 <td>MinTTY WSL1</td> 1345 <td>0.57</td> 1346 </tr> 1347 <tr> 1348 <td>MinTTY Cygwin</td> 1349 <td>2.2</td> 1350 </tr> 1351 <tr> 1352 <td>Windows Terminal</td> 1353 <td>5.25</td> 1354 </tr> 1355 <tr> 1356 <td>Alacritty</td> 1357 <td>5.75</td> 1358 </tr> 1359 <tr> 1360 <td>WezTerm</td> 1361 <td>6.2</td> 1362 </tr> 1363 <tr> 1364 <td>conhost.exe</td> 1365 <td>21.8</td> 1366 </tr> 1367 </tbody> 1368</table> 1369 1370<p>I assume this means MinTTY throttles display updates in some way. Of 1371course this is totally fine, because you couldnât read the output 1372either way.</p> 1373 1374<p>To test the hypothesis that MinTTY was caching cell rendering by 1375their contents, I also tried generating a file that rotated through 1376different lines, with no effect.</p> 1377 1378<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s">"/tmp/lines2.txt"</span><span class="p">,</span> <span class="s">"w"</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span> 1379 <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">100000</span><span class="p">):</span> 1380 <span class="n">sentence</span><span class="o">=</span><span class="s">"This sentence has forty-five (45) characters."</span> 1381 <span class="k">print</span><span class="p">(</span><span class="n">sentence</span><span class="p">[</span><span class="n">i</span><span class="o">%</span><span class="nb">len</span><span class="p">(</span><span class="n">sentence</span><span class="p">):]</span><span class="o">+</span><span class="n">sentence</span><span class="p">[:</span><span class="n">i</span><span class="o">%</span><span class="nb">len</span><span class="p">(</span><span class="n">sentence</span><span class="p">)],</span> <span class="nb">file</span><span class="o">=</span><span class="n">f</span><span class="p">)</span> 1382</code></pre></div></div> 1383 1384<h3 id="cpu-usage-during-repeated-keypresses">CPU Usage During Repeated Keypresses</h3> 1385 1386<p>While making these measurements, I noticed some strange behaviors. My 1387monitor runs at 120 Hz and animation and window dragging are generally 1388smooth. But right after you start Alacritty, dragging the window 1389animates at something like 30-60 frames per second. Itâs noticeably 1390chunkier. WezTerm does the same, but slightly worse. Maybe 20 frames 1391per second.</p> 1392 1393<p>I donât know if I can blame the terminals themselves, because I 1394sometimes experience this even with Notepad.exe too. But the 1395choppiness stands out much more. Maybe something is CPU-bound in 1396responding to window events?</p> 1397 1398<p>This made me think of a new test: if I open a terminal and hold down 1399the âaâ button on autorepeat, how much CPU does the terminal consume?</p> 1400 1401<p>To measure this, I set the terminal processâs affinity to my third 1402physical core, and watched the CPU usage graph in Task Manager. Not a 1403great methodology, but it gave a rough sense. Again, 80x50.</p> 1404 1405<table> 1406 <thead> 1407 <tr> 1408 <th>Terminal</th> 1409 <th>Percent of Core</th> 1410 <th>Private Bytes After Startup (KiB)</th> 1411 </tr> 1412 </thead> 1413 <tbody> 1414 <tr> 1415 <td>conhost</td> 1416 <td>0%</td> 1417 <td>6,500</td> 1418 </tr> 1419 <tr> 1420 <td>Alacritty</td> 1421 <td>5%</td> 1422 <td>74,000</td> 1423 </tr> 1424 <tr> 1425 <td>MinTTY WSL1</td> 1426 <td>10%</td> 1427 <td>10,200</td> 1428 </tr> 1429 <tr> 1430 <td>MinTTY Cygwin</td> 1431 <td>10%</td> 1432 <td>10,500</td> 1433 </tr> 1434 <tr> 1435 <td>Windows Terminal</td> 1436 <td>20%</td> 1437 <td>73,700</td> 1438 </tr> 1439 <tr> 1440 <td>WezTerm</td> 1441 <td>85%</td> 1442 <td>134,000</td> 1443 </tr> 1444 </tbody> 1445</table> 1446 1447<p>The WezTerm CPU usage has to be a bug. Iâll report it.</p> 1448 1449<h3 id="cpu-usage-idle">CPU Usage (Idle)</h3> 1450 1451<p>I often have a pile of idle terminals sitting around. I donât want 1452them to chew battery life. So letâs take a look at CPU Cycles Delta 1453(courtesy of Process Explorer) with a fresh, idle WSL 1454session.</p> 1455 1456<table> 1457 <thead> 1458 <tr> 1459 <th>Terminal</th> 1460 <th>Idle Cycles/s (Focused)</th> 1461 <th>
1461Idle Cycles/s (Background)</th> 1462 </tr> 1463 </thead> 1464 <tbody> 1465 <tr> 1466 <td>conhost</td> 1467 <td>~900,000</td> 1468 <td>0</td> 1469 </tr> 1470 <tr> 1471 <td>Alacritty</td> 1472 <td>~2,400,000</td> 1473 <td>no difference</td> 1474 </tr> 1475 <tr> 1476 <td>WezTerm</td> 1477 <td>~2,600,000</td> 1478 <td>~1,600,000</td> 1479 </tr> 1480 <tr> 1481 <td>Windows Terminal</td> 1482 <td>~55,000,000</td> 1483 <td>~6,100,000</td> 1484 </tr> 1485 <tr> 1486 <td>MinTTY WSL1</td> 1487 <td>~120,000,000</td> 1488 <td>no difference</td> 1489 </tr> 1490 <tr> 1491 <td>MinTTY Cygwin</td> 1492 <td>~120,000,000</td> 1493 <td>no difference</td> 1494 </tr> 1495 </tbody> 1496</table> 1497 1498<p>These numbers arenât great at all! For perspective, I have a pile of 1499Firefox tabs open, some of them actively running JavaScript, and 1500theyâre âonlyâ using a few hundred million cycles per second.</p> 1501 1502<p>Raymond Chen once wrote a <a href="https://devblogs.microsoft.com/oldnewthing/20060124-17/?p=32553">blog post about the importance of properly 1503idling</a> 1504in the Windows Terminal Server days. You might have a dozen users 1505logged into a host, and if a program is actively polling, itâs eating 1506performance that others could use.</p> 1507 1508<p>Today, we often run on batteries, so idling correctly still matters, 1509but it seems to be something of a lost art. The only terminal that 1510idles completely is the old conhost.exe.</p> 1511 1512<p>The other lesson we can draw is that Microsoftâs own replacement for 1513conhost.exe, Windows Terminal, uses over 10x the RAM, 60x the CPU when 1514focused, and infinitely more CPU when idle.</p> 1515 1516<h2 id="conclusions">Conclusions</h2> 1517 1518<p>conhost.exe consistently has the best latency, with MinTTY not much 1519behind. MinTTY handily dominates the throughput test, supports all 1520major ANSI character attributes, and has a better default palette.</p> 1521 1522<p>As in 2009, Iâd say MinTTY is still pretty great. (I should try to 1523track down that idle CPU consumption. It feels more like a bug than a 1524requirement.)</p> 1525 1526<p>If you want to use MinTTY as the default terminal for WSL, install 1527<a href="https://github.com/mintty/wsltty">WSLtty</a>.</p> 1528 1529<p>The others all have slightly worse latencies, but theyâre in a similar 1530class. Iâm particularly sensitive to latency, so Iâd had a suspicion 1531even before measuring. Maybe itâs some consequence of being 1532GPU-accelerated? Out of curiousity, I put Windows Terminal in 1533software-rendered mode, and it shaved perhaps 4 ms off (median of 62.5 1534ms, 15 frames). Perhaps just measurement noise.</p> 1535 1536<p>While Iâm going to stick with MinTTY, one thing is clear: there is 1537room to improve all of the above.</p> 1538 1539 </li><li><span class="post-meta">Feb 10, 2024</span> 1540 <h3> 1541 <a class="post-link" href="/2024/02/tmux-config/"> 1542 My Minimal tmux Config 1543 </a> 1544 </h3> 1545 <p>If you spend any significant time in a terminal, youâve probably used 1546<a href="https://github.com/tmux/tmux/wiki">tmux</a>.</p> 1547 1548<p>Iâm writing this post for a few reasons:</p> 1549<ul> 1550 <li>People have asked for my config.</li> 1551 <li>I see too many people wasting their time in the morning, rebuilding 1552their session from the previous day.</li> 1553 <li>I felt I should justify the configuration to myself rather than 1554setting options ad-hoc.</li> 1555</ul> 1556 1557<p>tmux is often paired with a persistent connection. There are two 1558popular choices: Eternal Terminal and Mosh. The goal is to close your 1559laptop at the end of the day, open it the next morning, and have 1560everything where it was so you can immediately get back into flow.</p> 1561 1562<p><strong>Note</strong>: There are other options. WezTerm has <a href="https://wezfurlong.org/wezterm/multiplexing.html">built-in 1563multiplexing</a>, for 1564example.</p> 1565 1566<h2 id="macos--iterm2--eternal-terminal--tmux-control-mode">macOS + iTerm2 + Eternal Terminal + tmux Control Mode</h2> 1567 1568<p>If you use macOS, <a href="https://iterm2.com/">iTerm2</a> has deep 1569<a href="https://iterm2.com/documentation-tmux-integration.html">tmux 1570integration</a> 1571in the form of <a href="https://github.com/tmux/tmux/wiki/Control-Mode">tmux Control 1572Mode</a>.</p> 1573 1574<p>tmux windows map to iTerm2 tabs. Native tab navigation and scrollback 1575(both scrolling and find) work just as youâd expect.</p> 1576 1577<p>tmux control mode does expect a reliable stream channel, so if you 1578want a connection that persists even when network connections are 1579dropped, Mosh will not work. Youâll need <a href="https://eternalterminal.dev/">Eternal 1580Terminal</a>.</p> 1581 1582<p>If you use a Mac, this is an excellent configuration. I worked on 1583<a href="https://github.com/facebook/sapling/blob/main/eden/fs/docs/Overview.md">EdenFS</a> 1584and <a href="https://facebook.github.io/watchman/">Watchman</a> for almost five 1585years this way.</p> 1586 1587<h2 id="mosh--tmux">mosh + tmux</h2> 1588 1589<p>But now I use Windows and Linux and canât use iTerm2, so tmux within 1590<a href="https://mosh.org/">Mosh</a> it is.</p> 1591 1592<p>I find tmuxâs default keybindings a little awkward, and the colors 1593simultaneously harsh and too minimal, so I made a configuration to 1594match my tastes.</p> 1595 1596<p>You can <a href="https://gist.github.com/chadaustin/d4696e18217a7de9b2671549abcb54c4">download the full .tmux.conf 1597here</a>.</p> 1598 1599<p>(You can go crazy, but avoiding too much fanciness was my goal. If you 1600want all the bling, install the <a href="https://github.com/tmux-
1600plugins/tpm?tab=readme-ov-file">Tmux Plugin 1601Manager</a> and 1602things like <a href="https://github.com/erikw/tmux-powerline">tmux-powerline</a>.</p> 1603 1604<h2 id="tmuxconf">.tmux.conf</h2> 1605 1606<p>First, a small demonstration.</p> 1607 1608<div style="max-width: 500px">
1608<script async="" id="asciicast-xYtwktNN10OIgwImlFUviz5tr" src="https://asciinema.org/a/xYtwktNN10OIgwImlFUviz5tr.js"></script>
1608</div> 1609 1610<p>Despite all of the work I put into <a href="https://chadaustin.me/2024/01/truecolor-terminal-emacs/">my recent post about 24-bit color 1611in 1612terminals</a>, I 1613do still use some with limited color support. macOSâs Terminal.app 1614only supports the 256-color palette, and the Linux console only really 1615supports 8. The following selects the correct <code class="language-plaintext highlighter-rouge">tmux</code> terminfo entry.</p> 1616 1617<div class="language-conf highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Detect the correct TERM value for new sessions. 1618# if-shell uses /bin/sh, so bashisms like [[ do not work. 1619</span><span class="n">if</span> <span class="s2">"[ $(tput colors) = 16777216 ]"</span> { 1620 <span class="n">set</span> -<span class="n">g</span> <span class="n">default</span>-<span class="n">terminal</span> <span class="s2">"tmux-direct"</span> 1621} { 1622 <span class="n">if</span> <span class="s2">"[ $(tput colors) = 256 ]"</span> { 1623 <span class="n">set</span> -<span class="n">g</span> <span class="n">default</span>-<span class="n">terminal</span> <span class="s2">"tmux-256color"</span> 1624 } { 1625 <span class="n">set</span> -<span class="n">g</span> <span class="n">default</span>-<span class="n">terminal</span> <span class="s2">"tmux"</span> 1626 } 1627} 1628</code></pre></div></div> 1629 1630<p>I prefer Emacs keybindings in both bash (readline) and tmux.</p> 1631 1632<div class="language-conf highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">setw</span> -<span class="n">g</span> <span class="n">mode</span>-<span class="n">keys</span> <span class="n">emacs</span> 1633</code></pre></div></div> 1634 1635<p>The next setting is more legacy terminal insanity. On some (most?) 1636terminals, programs cannot differentiate between a user pressing the 1637escape key and the beginning of an escape sequence. <code class="language-plaintext highlighter-rouge">readline</code> and 1638<code class="language-plaintext highlighter-rouge">tmux</code> default to 500 ms, which adds <a href="https://unix.stackexchange.com/a/608179/459183">noticeable 1639latency</a> in some 1640terminals when using programs like <code class="language-plaintext highlighter-rouge">vi</code>.</p> 1641 1642<p>Thereâs no correct value here. Ideally, your terminal would use an 1643unambiguous code for the escape key, <a href="https://github.com/mintty/mintty/wiki/CtrlSeqs#escape-keycode">like 1644MinTTY</a>.</p> 1645 1646<div class="language-conf highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">set</span> -<span class="n">s</span> <span class="n">escape</span>-<span class="n">time</span> <span class="m">200</span> 1647</code></pre></div></div> 1648 1649<p>Letâs not be stingy with scrollback! Searching lots of history is 1650worth spending megabytes.</p> 1651 1652<div class="language-conf highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># I can afford 50 MB of scrollback. 1653# Measured on WSL 1 with: 1654# yes $(python3 -c "print('y' * 80)") 1655</span><span class="n">set</span> -<span class="n">g</span> <span class="n">history</span>-<span class="n">limit</span> <span class="m">100000</span> 1656</code></pre></div></div> 1657 1658<p>By default, if multiple clients connect to one tmux session, tmux will 1659resize all of the windows to the smallest connected terminal.</p> 1660 1661<p>This behavior is annoying, and itâs always an accident. Sometimes Iâll 1662leave a temporary connection to a server from home and then another 1663fullscreen connection from work will cram each window into 80x25.</p> 1664 1665<p>The <code class="language-plaintext highlighter-rouge">aggressive-resize</code> option applies this logic only to the 1666currently-viewed window, not everything in the session.</p> 1667 1668<div class="language-conf highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">setw</span> -<span class="n">g</span> <span class="n">aggressive</span>-<span class="n">resize</span> <span class="n">on</span> 1669</code></pre></div></div> 1670 1671<p>Window titles donât automatically forward to the whatever graphical 1672terminal youâre using. Do that, and add the hostname, but keep it 1673concise.</p> 1674 1675<div class="language-conf highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">set</span> -<span class="n">g</span> <span class="n">set</span>-<span class="n">titles</span> <span class="n">on</span> 1676<span class="n">set</span> -<span class="n">g</span> <span class="n">set</span>-<span class="n">titles</span>-<span class="n">string</span> <span class="s2">"#h: #W"</span> 1677</code></pre></div></div> 1678 1679<p>iTerm2 has this nice behavior where active tabs are visually marked so 1680you can see, at a glance, which had recent activity. The following two 1681options offer similar behavior. Setting <code class="language-plaintext highlighter-rouge">activity-action</code> to <code class="language-plaintext highlighter-rouge">none</code> 1682disables any audible ding or visible flash, leaving just a subtle 1683indication in the status bar.</p> 1684 1685<div class="language-conf highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">set</span> -<span class="n">g</span> <span class="n">
1685monitor</span>-<span class="n">activity</span> <span class="n">on</span> 1686<span class="n">set</span> -<span class="n">g</span> <span class="n">activity</span>-<span class="n">action</span> <span class="n">none</span> 1687</code></pre></div></div> 1688 1689<p>The following is perhaps the most important part of my configuration: 1690tab management. Like browsers and iTerm2, I want my tabs numbered. I 1691want a single (modified) keypress to select a tab, and I want tabs 1692automatically renumbered as theyâre created, destroyed, and reordered.</p> 1693 1694<p>I also want iTerm2-style previous- and next-tab keybindings.</p> 1695 1696<div class="language-conf highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Match window numbers to the order of the keys on a keyboard. 1697</span><span class="n">set</span> -<span class="n">g</span> <span class="n">base</span>-<span class="n">index</span> <span class="m">1</span> 1698<span class="n">setw</span> -<span class="n">g</span> <span class="n">pane</span>-<span class="n">base</span>-<span class="n">index</span> <span class="m">1</span> 1699 1700<span class="n">setw</span> -<span class="n">g</span> <span class="n">renumber</span>-<span class="n">windows</span> <span class="n">on</span> 1701 1702<span class="c"># My tmux muscle memory still wants C-b 0 to select the first window. 1703</span><span class="n">bind</span> <span class="m">0</span> <span class="n">select</span>-<span class="n">window</span> -<span class="n">t</span> <span class="s2">":^"</span> 1704<span class="c"># Other terminals and web browsers use 9 to focus the final tab. 1705</span><span class="n">bind</span> <span class="m">9</span> <span class="n">select</span>-<span class="n">window</span> -<span class="n">t</span> <span class="s2">":$"</span> 1706 1707<span class="n">bind</span> -<span class="n">n</span> <span class="s2">"M-0"</span> <span class="n">select</span>-<span class="n">window</span> -<span class="n">t</span> <span class="s2">":^"</span> 1708<span class="n">bind</span> -<span class="n">n</span> <span class="s2">"M-1"</span> <span class="n">select</span>-<span class="n">window</span> -<span class="n">t</span> <span class="s2">":1"</span> 1709<span class="n">bind</span> -<span class="n">n</span> <span class="s2">"M-2"</span> <span class="n">select</span>-<span class="n">window</span> -<span class="n">t</span> <span class="s2">":2"</span> 1710<span class="n">bind</span> -<span class="n">n</span> <span class="s2">"M-3"</span> <span class="n">select</span>-<span class="n">window</span> -<span class="n">t</span> <span class="s2">":3"</span> 1711<span class="n">bind</span> -<span class="n">n</span> <span class="s2">"M-4"</span> <span class="n">select</span>-<span class="n">window</span> -<span class="n">t</span> <span class="s2">":4"</span> 1712<span class="n">bind</span> -<span class="n">n</span> <span class="s2">"M-5"</span> <span class="n">select</span>-<span class="n">window</span> -<span class="n">t</span> <span class="s2">":5"</span> 1713<span class="n">bind</span> -<span class="n">n</span> <span class="s2">"M-6"</span> <span class="n">select</span>-<span class="n">window</span> -<span class="n">t</span> <span class="s2">":6"</span> 1714<span class="n">bind</span> -<span class="n">n</span> <span class="s2">"M-7"</span> <span class="n">select</span>-<span class="n">window</span> -<span class="n">t</span> <span class="s2">":7"</span> 1715<span class="n">bind</span> -<span class="n">n</span> <span class="s2">"M-8"</span> <span class="n">select</span>-<span class="n">window</span> -<span class="n">t</span> <span class="s2">":8"</span> 1716<span class="c"># Browsers also select last tab with M-9. 1717</span><span class="n">bind</span> -<span class="n">n</span> <span class="s2">"M-9"</span> <span class="n">select</span>-<span class="n">window</span> -<span class="n">t</span> <span class="s2">":$"</span> 1718<span class="c"># Match iTerm2. 1719</span><span class="n">bind</span> -<span class="n">n</span> <span class="s2">"M-{"</span> <span class="n">previous</span>-<span class="n">window</span> 1720<span class="n">bind</span> -<span class="n">n</span> <span class="s2">"M-}"</span> <span class="n">next</span>-<span class="n">window</span> 1721</code></pre></div></div> 1722 1723<p>Note that Emacs assigns meaning to Alt-number. If it matters to you, 1724pick a different modifier.</p> 1725 1726<p>Now letâs optimize the window ordering. By default, <code class="language-plaintext highlighter-rouge">C-b c</code> creates a 1727new window at the end. Thatâs a fine default. But sometimes I want a 1728new window right after the current one, so define <code class="language-plaintext highlighter-rouge">C-b C-c</code>. Also, add 1729some key bindings for sliding the current window around.</p> 1730 1731<div class="language-conf highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">bind</span> <span class="s2">"C-c"</span> <span class="n">new</span>-<span class="n">window</span> -<span class="n">a</span> 1732 1733<span class="n">bind</span> <span class="s2">"S-Left"</span> { 1734 <span class="n">swap</span>-<span class="n">window</span> -<span class="n">t</span> -<span class="m">1</span> 1735 <span class="n">select</span>-<span class="n">window</span> -<span class="n">t</span> -<span class="m">1</span> 1736} 1737<span class="n">bind</span> <span class="s2">"S-Right"</span> { 1738 <span class="n">swap</span>-<span class="n">window</span> -<span class="n">t</span> +<span class="m">1</span> 1739 <span class="n">select</span>-<span class="n">window</span> -<span class="n">t</span> +<span class="m">1</span> 1740} 1741</code></pre></div></div> 1742 1743<p>I wanted âC-{â and âC-}â but terminal key encoding <a href="https://vt100.net/docs/vt100-ug/chapter3.html">doesnât work like 1744that</a>.</p> 1745 1746<p>Next, letâs define some additional key bindings for very common 1747operations.</p> 1748 1749<p>By default, searching in the scrollback requires entering âcopy modeâ 1750with <code class="language-plaintext highlighter-rouge">C-b [</code> and then entering reverse search mode with <code class="language-plaintext highlighter-rouge">C-r</code>. 1751Searching is common, so give it a dedicated <code class="language-plaintext highlighter-rouge">C-b r</code>.</p> 1752 1753<div class="language-conf highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">bind</span> <span class="n">r</span> { 1754 <span class="n">copy</span>-<span class="n">mode</span> 1755 <span class="n">command</span>-<span class="n">prompt</span> -<span class="n">i</span> -<span class="n">p</span> <span class="s2">"(search up)"</span> \ 1756 <span class="s2">"send-keys -X search-backward-incremental '%%%'"</span> 1757} 1758</code></pre></div></div> 1759 1760<p>And some convenient toggles:</p> 1761 1762<div class="language-conf highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Toggle terminal mouse support. 1763</span><span class="n">bind</span> <span class="n">m</span> <span class="n">set</span>-<span class="n">option</span> -<span class="n">g</span> <span class="n">
1763mouse</span> \; <span class="n">display</span> <span class="s2">"Mouse: #{?mouse,ON,OFF}"</span> 1764 1765<span class="c"># Toggle status bar. Useful for fullscreen focus. 1766</span><span class="n">bind</span> <span class="n">t</span> <span class="n">set</span>-<span class="n">option</span> <span class="n">status</span> 1767</code></pre></div></div> 1768 1769<p>Now the status bar. The default status bar is okay, but we can do 1770better.</p> 1771 1772<figure> 1773<a href="/images/tmux/tmux-status-before.png"><img src="/images/tmux/tmux-status-before.png" alt="tmux status bar: before" /></a> 1774<figcaption>tmux status bar: before</figcaption> 1775</figure> 1776 1777<ul> 1778 <li>Move the tmux session ID next to the hostname on the right side.</li> 1779 <li>Move the current time to the far right corner.</li> 1780 <li>Keep the date, but I think I can remember what year it is.</li> 1781 <li>Ensure there is a single space between the windows and the left 1782edge. Without a space at the edge, it looks weird.</li> 1783</ul> 1784 1785<figure> 1786<a href="/images/tmux/tmux-status-after.png"><img src="/images/tmux/tmux-status-after.png" alt="tmux status bar: after" /></a> 1787<figcaption>tmux status bar: after</figcaption> 1788</figure> 1789 1790<p>The other half of that improvement is the color scheme. Instead of a 1791harsh black-on-green, I chose a scheme that evokes <a href="https://www.worthpoint.com/worthopedia/dec-digital-vt520-monitor-amber-1791664954">old amber CRT 1792phosphors</a> 1793or <a href="https://retropaq.com/the-miracle-of-gas-plasma/">gas plasma 1794displays</a> My dad had 1795a âlaptopâ with one of those when I was young.</p> 1796 1797<p>The following color scheme mildly highlights the current window and 1798uses a dark blue for the hostname-and-time section. These colors donât 1799distract me when Iâm not working, but if I do look, the important 1800information is there.</p> 1801 1802<div class="language-conf highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">if</span> <span class="s2">"[ $(tput colors) -ge 256 ]"</span> { 1803 <span class="n">set</span> -<span class="n">g</span> <span class="n">status</span>-<span class="n">left</span>-<span class="n">style</span> <span class="s2">"fg=black bg=colour130"</span> 1804 <span class="n">set</span> -<span class="n">g</span> <span class="n">status</span>-<span class="n">right</span>-<span class="n">style</span> <span class="s2">"bg=colour17 fg=orange"</span> 1805 <span class="n">set</span> -<span class="n">g</span> <span class="n">status</span>-<span class="n">style</span> <span class="s2">"fg=black bg=colour130"</span> 1806 <span class="n">set</span> -<span class="n">g</span> <span class="n">message</span>-<span class="n">style</span> <span class="s2">"fg=black bg=colour172"</span> 1807 <span class="c"># Current window should be slightly brighter. 1808</span> <span class="n">set</span> -<span class="n">g</span> <span class="n">window</span>-<span class="n">status</span>-<span class="n">current</span>-<span class="n">style</span> <span class="s2">"fg=black bg=colour172"</span> 1809 <span class="c"># Windows with activity should be very subtly highlighted. 1810</span> <span class="n">set</span> -<span class="n">g</span> <span class="n">window</span>-<span class="n">status</span>-<span class="n">activity</span>-<span class="n">style</span> <span class="s2">"fg=colour17 bg=colour130"</span> 1811 <span class="n">set</span> -<span class="n">g</span> <span class="n">mode</span>-<span class="n">style</span> <span class="s2">"fg=black bg=colour172"</span> 1812} 1813</code></pre></div></div> 1814 1815<p>And thatâs it!</p> 1816 1817<p>Again, feel free to copy <a href="https://gist.github.com/chadaustin/d4696e18217a7de9b2671549abcb54c4">the complete 1818.tmux.conf</a>.</p> 1819 1820<h2 id="shell-integration">Shell Integration</h2> 1821 1822<p>Thereâs one more config to mention: adding some shell aliases to 1823.bashrc.</p> 1824 1825<p>I sometimes want to look at or edit a file right next to my shell.</p> 1826 1827<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">if</span> <span class="o">[[</span> <span class="s2">"</span><span class="nv">$TMUX</span><span class="s2">"</span> <span class="o">]]</span><span class="p">;</span> <span class="k">
1827then 1828 function </span>lv<span class="o">()</span> <span class="o">{</span> 1829 tmux split-window <span class="nt">-h</span> less <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span> 1830 <span class="o">}</span> 1831 <span class="k">function </span>ev<span class="o">()</span> <span class="o">{</span> 1832 tmux split-window <span class="nt">-h</span> emacs <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span> 1833 <span class="o">}</span> 1834 <span class="k">function </span>lh<span class="o">()</span> <span class="o">{</span> 1835 tmux split-window <span class="nt">-v</span> less <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span> 1836 <span class="o">}</span> 1837 <span class="k">function </span>eh<span class="o">()</span> <span class="o">{</span> 1838 tmux split-window <span class="nt">-v</span> emacs <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span> 1839 <span class="o">}</span> 1840<span class="k">fi</span> 1841</code></pre></div></div> 1842 1843<p>(You may notice the aliases use different meanings of horizontal and 1844vertical than tmux. I donât know, it feels like tmux is backwards, but 1845that could be my brain.)</p> 1846
1847<script async="" id="asciicast-02hGqCVoOv13lpzI3XYZ1Tpz6" src="https://asciinema.org/a/02hGqCVoOv13lpzI3XYZ1Tpz6.js"></script>
1847 1848 1849<p>Happy multiplexing!</p> 1850 1851 </li><li><span class="post-meta">Jan 27, 2024</span> 1852 <h3> 1853 <a class="post-link" href="/2024/01/truecolor-terminal-emacs/"> 1854 I Just Wanted Emacs to Look Nice â Using 24-Bit Color in Terminals 1855 </a> 1856 </h3> 1857 <p>Thanks to some coworkers and David Wilsonâs <a href="https://www.youtube.com/watch?v=74zOY-vgkyw&list=PLEoMzSkcN8oPH1au7H6B7bBJ4ZO7BXjSZ">Emacs from Scratch 1858playlist</a>, 1859Iâve been getting back into Emacs. The community is more vibrant than 1860the last time I looked, and 1861<a href="https://microsoft.github.io/language-server-protocol/">LSP</a> brings 1862modern completion and inline type checking.</p> 1863 1864<p>Davidâs Emacs looks so fancy â I want nice colors and fonts 1865too, especially my preferred themes like 1866<a href="https://ethanschoonover.com/solarized/">Solarized</a>.</p> 1867 1868<p>From desktop environments, Emacs automatically supports 24-bit color.</p> 1869 1870<figure> 1871<a href="/images/truecolor-terminal-emacs/emacs-window.png"><img src="/images/truecolor-terminal-emacs/emacs-window.png" alt="Graphical Emacs: Fonts and Colors" /></a> 1872<figcaption>Graphical Emacs: Fonts and Colors</figcaption> 1873</figure> 1874 1875<p>But, since I work on infrastructure, Iâve lived primarily in terminals 1876for years. And my Emacs looks like:</p> 1877 1878<figure> 1879<a href="/images/truecolor-terminal-emacs/emacs-terminal.png"><img src="/images/truecolor-terminal-emacs/emacs-terminal.png" alt="Terminal Emacs: Not Fancy" /></a> 1880<figcaption>Terminal Emacs: Not Fancy</figcaption> 1881</figure> 1882 1883<p>It turns out, for <em>years</em>
1883, <a href="https://github.com/termstandard/colors#truecolor-support-in-output-devices">popular terminals have supported 24-bit 1884color</a>. 1885And yet theyâre rarely used.</p> 1886 1887<p>Like everything else, it boil down to legacy and politics. Control 1888codes are a protocol, and changes to that protocol take time to 1889propagate, especially with missteps along the way.</p> 1890 1891<p>This post is two things:</p> 1892<ol> 1893 <li>how to enable true-color support in the terminal environments I 1894use, and</li> 1895 <li>how my desire for nice colors in Emacs led to poring over technical 1896standards from the 70s, 80s, and 90s, wondering how we got to this 1897point.</li> 1898</ol> 1899 1900<blockquote> 1901 <p><strong><em>NOTE:</em></strong> I did my best, but please forgive any terminology 1902slip-ups or false histories. I grew up on VGA text mode UIs, but 1903never used a hardware terminal and wasnât introduced to unix until 1904much later.</p> 1905</blockquote> 1906 1907<h2 id="ansi-escape-codes">ANSI Escape Codes</h2> 1908 1909<p>Early hardware terminals offered their own, incompatible, control code 1910schemes. That made writing portable software hard, so ANSI 1911standardized the protocol, while reserving room for expansion and 1912vendor-specific capabilities.</p> 1913 1914<figure> 1915<a href="/images/truecolor-terminal-emacs/dec-vt100.webp"><img src="/images/truecolor-terminal-emacs/dec-vt100.webp" alt="DEC 1916VT100 (1978)" /></a> 1917<figcaption>DEC 1918VT100 (1978)</figcaption> 1919</figure> 1920 1921<p><a href="https://en.wikipedia.org/wiki/ANSI_escape_code">ANSI escape codes</a> 1922date back to the 70s. They cover a huge range of functionality, but 1923since this post is focused on colors, Iâm mostly interested in SGR 1924(Select Graphics Rendition), which allows configuring a variety of 1925character display attributes:</p> 1926 1927<ul> 1928 <li>bold or intensity</li> 1929 <li>italics (not frequently supported)</li> 1930 <li>blink</li> 1931 <li>foreground and background colors</li> 1932 <li>and a bunch of other stuff. You can look at Wikipedia.</li> 1933</ul> 1934 1935<h2 id="3--4--and-8-bit-color">3-, 4-, and 8-bit Color</h2> 1936 1937<p>When color was introduced, there were eight. Black, white, the 1938additive primaries, and the subtractive primaries. The eight corners 1939of an RGB color cube.</p> 1940 1941<p>Later, a bright (or bold) bit added eight more; âbright blackâ being 1942dark gray.</p> 1943 1944<figure> 1945<a href="/images/truecolor-terminal-emacs/microsoft-vga.png"><img src="/images/truecolor-terminal-emacs/microsoft-vga.png" alt="4-Bit VGA Text Mode Palette" /></a> 1946<figcaption>4-Bit VGA Text Mode Palette</figcaption> 1947</figure> 1948 1949<p>In 1999, <a href="https://invisible-island.net/xterm/xterm.log.html#xterm_111">Todd Larason patched xterm to add support for 256 1950colors</a>. 1951He chose a palette that filled out the RGB color cube with a 6x6x6 1952interior sampling and added a 24-entry finer-precision grayscale ramp.</p> 1953 1954<figure> 1955<a href="/images/truecolor-terminal-emacs/xterm-256color.png"><img src="/images/truecolor-terminal-emacs/xterm-256color.png" alt="Output From colortest-256" /></a> 1956<figcaption>Output From colortest-256</figcaption> 1957</figure> 1958 1959<blockquote> 1960 <p><strong><em>NOTE:</em></strong> Thereâs a rare, but still-supported, 88-color variant 1961with a 4x4x4 color cube and 8-entry grayscale ramp, primarily to 1962reduce the use of historically-limited X11 color objects.</p> 1963</blockquote> 1964 1965<blockquote> 1966 <p><strong><em>NOTE:</em></strong> Weâll debug this later, but Toddâs patch to add 1967256-color support to xterm used semicolons as the separator between 1968the ANSI SGR command 48 and the color index, which set off a chain 1969reaction of ambiguity weâre still dealing with today.</p> 1970</blockquote> 1971 1972<h2 id="where-did-24-bit-color-support-come-from">Where Did 24-Bit Color Support Come From?</h2> 1973 1974<p>Itâs well-documented how to send 8-bit and 24-bit colors to compatible 1975terminals. Per 1976<a href="https://en.wikipedia.org/wiki/ANSI_escape_code#8-bit">Wikipedia</a>:</p> 1977 1978<p><code class="language-plaintext highlighter-rouge">ESC[38;5;<n>m</code> sets foreground color <code class="language-plaintext highlighter-rouge">n</code> per the palettes above.</p> 1979 1980<p><code class="language-plaintext highlighter-rouge">ESC[38;2;<r>;<g>;<b>m</code> sets foreground color (<code class="language-plaintext highlighter-rouge">r</code>, <code class="language-plaintext highlighter-rouge">g</code>, <code class="language-plaintext highlighter-rouge">b</code>).</p> 1981 1982<p>(Again, that confusion about <a href="https://wezfurlong.org/wezterm/escape-sequences.html#graphic-rendition-sgr">semicolons vs. 1983colons</a>, 1984and an unused colorspace ID if colons are used. Weâll get to the 1985bottom of that soon.)</p> 1986 1987<p>But why 5? Why 2? How did any of this come about? Iâd struggled enough 1988with unexpected output that it was time to discover the ground truth.</p> 1989 1990<p>Finding and reading original sources led me to construct the following 1991narrative:</p> 1992 1993<ul> 1994 <li>In the 70s, ANSI standardized terminal escape sequences, resulting 1995in <a href="https://nvlpubs.nist.gov/nistpubs/Legacy/FIPS/fipspub86.pdf">ANSI 1996X3.64</a> 1997and the better-known 1998<a href="https://www.ecma-international.org/wp-content/uploads/ECMA-48_5th_edition_june_1991.pdf">ECMA-48</a>.</li> 1999 <li>The first edition of ECMA-48 is lost to time, but it probably looks 2000much like ANSI X3.64.</li> 2001 <li>The <a href="https://ecma-international.org/wp-content/uploads/ECMA-48_2nd_edition_august_1979.pdf">2nd 2002edition</a> 2003of ECMA-48 (1979) allocated SGR parameters 30-37 and 40-47 for setting 20043-bit foreground and background colors, respectively. 2005 <ul> 2006 <li>By the way, these standards use the word âparameterâ to mean 2007command, and âsubparameterâ to mean argument, if applicable.</li> 2008 </ul> 2009 </li> 2010 <li>The <a href="https://ecma-international.org/wp-content/uploads/ECMA-48_3rd_edition_march_1984.pdf">3rd 2011edition</a> 2012(1984) introduced the concept of an implementation-defined default 2013color for both foreground and background, and allocated parameters 201439 and 49, respectively.</li> 2015 <li>Somewhere in this timeline, vendors did ship hardware terminals with 2016richer color support. The <a href="https://terminals-wiki.org/wiki/index.php/Wyse_WY-370">Wyse 2017WY-370</a> 2018introduced new color modes, including a direct-indexed 64-color
2019palette. (See Page 86 of its <a href="http://bitsavers.org/pdf/wyse/WY-370/881133-02A_WY-370_Programmers_Guide_Jun90.pdf">Programmerâs 2020Guide</a>.)</li> 2021 <li>38 and 48 are the most important parameters for selecting colors 2022today, but they werenât allocated by either the 2023<a href="https://ecma-international.org/wp-content/uploads/ECMA-48_4th_edition_december_1986.pdf">4th</a> 2024(1986) or 2025<a href="https://www.ecma-international.org/wp-content/uploads/ECMA-48_5th_edition_june_1991.pdf">5th</a> 2026(1991) editions. So where did they come from? The 5th edition gives 2027a clue: 2028 <blockquote> 2029 <p>reserved for future standardization; intended for setting 2030character foreground colour as specified in ISO 8613-6 [CCITT 2031Recommendation T.416]</p> 2032 </blockquote> 2033 </li> 2034 <li> 2035 <p>ISO 8613 was a boondoggle of a project intended to <a href="https://en.wikipedia.org/wiki/Open_Document_Architecture">standardize and 2036replace all proprietary document file 2037formats</a>. 2038Youâve never heard of it, so it obviously failed. But its legacy 2039lives on â ISO 8613-6 (ITU T.416) (1993) built on ECMA-48âs codes 2040and defined parameters 38 and 48 as extended foreground and 2041background color modes, respectively.</p> 2042 2043 <blockquote> 2044 <p>The first parameter element indicates a choice between:</p> 2045 <ul> 2046 <li>0 implementation defined (only applicable for the character foreground colour)</li> 2047 <li>1 transparent;</li> 2048 <li>2 direct colour in RGB space;</li> 2049 <li>3 direct colour in CMY space;</li> 2050 <li>4 direct colour in CMYK space;</li> 2051 <li>5 indexed colour.</li> 2052 </ul> 2053 </blockquote> 2054 </li> 2055</ul> 2056 2057<p>There we go! <em>That</em> is why 5 is used for 256-color mode and 2 is 205824-bit RGB.</p> 2059 2060<p>Careful reading also gives a clue as to the semicolon vs. colon syntax 2061screw-up. Note the subtle use of the term âparameter elementâ vs. 2062âparameterâ.</p> 2063 2064<p>If you read ISO 8613-6 (ITU T.416) and ECMA-48 closely, itâs not 2065explicitly stated, but they seem to indicate that unknown parameters 2066for commands like âselect graphics renditionâ should be ignored. And 2067parameters are separated with semicolons.</p> 2068 2069<p>That implies <code class="language-plaintext highlighter-rouge">ESC[38;5;3m</code> should be interpreted, in terminals that 2070donât support SGR 38, as âunknown, ignored (38)â, âblinking (5)â, and 2071âitalicized (3)â. The syntax should use colons to separate 2072sub-parameter components, but something got lost along the way.</p> 2073 2074<p>(Now, in practice, programs are told how to communicate with their 2075terminals via the TERM variable and the terminfo database, so I 2076donât know how much pain occurs in reality.)</p> 2077 2078<p>Thomas Dickey has done a great job documenting the history of 2079<a href="https://invisible-island.net/ncurses/">ncurses</a> and 2080<a href="https://invisible-island.net/xterm/xterm.log.html">xterm</a>, and, lo 2081and behold, explains exactly the <a href="https://invisible-island.net/xterm/xterm.faq.html#semicolon_vs_colon">origin of the ambiguous 2082syntax</a>:</p> 2083 2084<blockquote> 2085 <p>We used semicolon (like other SGR parameters) for separating the 2086R/G/B values in the escape sequence, since a copy of ITU T.416 2087(ISO-8613-6) which presumably clarified the use of colon for this 2088feature was costly.</p> 2089 2090 <p>Using semicolon was incorrect because some applications could expect 2091their parameters to be order-independent. As used for the R/G/B 2092values, that was order-dependent. The relevant information, by the 2093way, is part of ECMA-48 (not ITU T.416, as mentioned in Why only 16 2094(or 256) colors?). Quoting from section 5.4.2 of ECMA-48, page 12, 2095and adding emphasis (not in the standard):</p> 2096 2097 <blockquote> 2098 <p>Each parameter sub-string consists of one or more bit combinations 2099from 03/00 to 03/10; the bit combinations from 03/00 to 03/09 2100represent the digits ZERO to NINE; bit combination 03/10 may be 2101used as a separator in a parameter sub-string, for example, to 2102separate the fractional part of a decimal number from the integer 2103part of that number.</p> 2104 </blockquote> 2105 2106 <p>and later on page 78, in 8.3.117 SGR â SELECT GRAPHIC RENDITION, the 2107description of SGR 38:</p> 2108 2109 <blockquote> 2110 <p>(reserved for future standardization; intended for setting 2111character foreground colour as specified in ISO 8613-6 [CCITT 2112Recommendation T.416])</p> 2113 </blockquote> 2114 2115 <p>Of course you will immediately recognize that 03/10 is ASCII colon, 2116and that ISO 8613-6 necessarily refers to the encoding in a
2117parameter sub-string. Or perhaps you will not.</p> 2118</blockquote> 2119 2120<p>So itâs all because the ANSI and ISO standards are ridiculously 2121expensive (to this day, these crappy PDF scans from the 90s and 2122earlier are $200 USD!) and because they use a baroque syntax to denote 2123ASCII characters. While writing this post, I had to keep <code class="language-plaintext highlighter-rouge">man ascii</code> 2124open to match, for example, <code class="language-plaintext highlighter-rouge">03/10</code> to colon and <code class="language-plaintext highlighter-rouge">03/11</code> to semicolon. 2125I guess itâs how standards were written back then. A Hacker News 2126thread in the context of WezTerm <a href="https://news.ycombinator.com/item?id=35138390">gives more 2127detail</a>.</p> 2128 2129<p>So, to recap in the timeline:</p> 2130 2131<ul> 2132 <li>1999: <a href="https://invisible-island.net/xterm/xterm.log.html#xterm_111">Thomas Dickey merged Todd Larasonâs 256-color 2133patches</a> 2134with ambiguous semicolon syntax.</li> 2135 <li>2006: Konsole added support for 256-color and 24-bit truecolor using 2136the same ambiguous syntax as xterm, with a <a href="https://bugs.kde.org/show_bug.cgi?id=107487">follow-on 2137discussion</a> about 2138colons vs. semicolons. The issue was noticed, but semicolon syntax 2139was adopted anyway.</li> 2140 <li>2012: Thomas Dickey <a href="https://invisible-island.net/xterm/xterm.log.html#xterm_282">fixed xterm to accept the standards-compliant 2141syntax</a>.</li> 2142 <li>2016: Windows 10âs built-in console gained <a href="https://devblogs.microsoft.com/commandline/24-bit-color-in-the-windows-console/">ANSI escape code 2143support, including 24-bit 2144colors</a>. 2145Unfortunately with the ambiguous semicolon syntax.</li> 2146 <li>2019: Windows Terminal is released, with ANSI escape code support, 2147but also using ambiguous semicolon syntax.</li> 2148 <li>2022: Microsoft announced <a href="https://learn.microsoft.com/en-us/windows/console/ecosystem-roadmap">ecosystem-wide 2149migration</a> 2150from the legacy framebuffer-based VGA-style console subsystem to 2151ANSI terminal emulation, specifically using xterm as a guide.</li> 2152 <li>2022: Konsole <a href="https://github.com/KDE/konsole/commit/316a386d92a083e235624e9f81df3b6dbbe08bff">gains support for standards-compliant 2153syntax</a>.</li> 2154</ul> 2155 2156<p>Okay, hereâs what weâve established:</p> 2157 2158<ul> 2159 <li>ANSI codes are widely supported, even on Windows.</li> 2160 <li>Truecolor support is either widely supported or (for example, on the 2161Linux text mode terminal) at least recognized and mapped to a more 2162limited palette.</li> 2163 <li>Semicolon syntax is the most compatible, though the unambiguous 2164colon syntax is slowly spreading.</li> 2165</ul> 2166 2167<p>I wrote a <a href="https://gist.github.com/chadaustin/2d2c2cb4b71fd1d4163aa8115077624a">small colortest.rs program to test color support and attributes like 2168reverse and 2169italics</a> 2170to confirm the above in every terminal I use.</p> 2171 2172<h2 id="terminfo">Terminfo</h2> 2173 2174<p>Now that weâve established terminal capabilities and how to use them, 2175the next trick is to convince software of varying lineages to detect 2176and use the best color support available.</p> 2177 2178<p>Typically, this is done with the old 2179<a href="https://en.wikipedia.org/wiki/Terminfo">terminfo</a> library (or the 2180even older <a href="https://en.wikipedia.org/wiki/Termcap">termcap</a>).</p> 2181 2182<p>Terminfo provides a database of terminal capabilities and the ability 2183to generate appropriate escape sequences. The TERM environment 2184variable tells programs which terminfo record to use. Its value is 2185automatically forwarded over <code class="language-plaintext highlighter-rouge">ssh</code> connections.</p> 2186 2187<p>Terminfo uses ridiculous command names: <code class="language-plaintext highlighter-rouge">infocmp</code>, <code class="language-plaintext highlighter-rouge">tic</code>, <code class="language-plaintext highlighter-rouge">toe</code>. (Not 2188to be confused with the unrelated <code class="language-plaintext highlighter-rouge">tac</code>.)</p> 2189 2190<p>To see the list of terminfo records installed on your host, run <code class="language-plaintext highlighter-rouge">toe 2191-a</code>. (Do we /really/ need to install support for every legacy hardware 2192terminal on modern machines? Good luck even finding a hardware 2193terminal these days. Theyâre collectorâs items.)</p> 2194 2195<p><code class="language-plaintext highlighter-rouge">infocmp</code> is how you inspect the capabilities of a specific terminfo 2196record.</p> 2197 2198<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ infocmp xterm-256color 2199# Reconstructed via infocmp from file: /lib/terminfo/x/xterm-256color 2200xterm-256color|xterm with 256 colors, 2201 am, bce, ccc, km, mc5i, mir, msgr, npc, xenl, 2202 colors#0x100, cols#80, it#8, lines#24, pairs#0x10000, 2203 acsc=``aaffggiijjkkllmmnnooppqqrrssttuuvvwwxxyyzz{{||}}~~, 2204 bel=^G, blink=\E[5m, bold=\E[1m, cbt=\E[Z, civis=\E[?25l, 2205 clear=\E[H\E[2J, cnorm=\E[?12l\E[?25h, cr=\r, 2206 csr=\E[%i%p1%d;%p2%dr, cub=\E[%p1%dD, cub1=^H,
2207 cud=\E[%p1%dB, cud1=\n, cuf=\E[%p1%dC, cuf1=\E[C, 2208 cup=\E[%i%p1%d;%p2%dH, cuu=\E[%p1%dA, cuu1=\E[A, 2209 cvvis=\E[?12;25h, dch=\E[%p1%dP, dch1=\E[P, dim=\E[2m, 2210 dl=\E[%p1%dM, dl1=\E[M, ech=\E[%p1%dX, ed=\E[J, el=\E[K, 2211 el1=\E[1K, flash=\E[?5h$<100/>\E[?5l, home=\E[H, 2212 hpa=\E[%i%p1%dG, ht=^I, hts=\EH, ich=\E[%p1%d@, 2213 il=\E[%p1%dL, il1=\E[L, ind=\n, indn=\E[%p1%dS, 2214 initc=\E]4;%p1%d;rgb:%p2%{255}%*%{1000}%/%2.2X/%p3%{255}%*%{1000}%/%2.2X/%p4%{255}%*%{1000}%/%2.2X\E\\, 2215 invis=\E[8m, is2=\E[!p\E[?3;4l\E[4l\E>, kDC=\E[3;2~, 2216 kEND=\E[1;2F, kHOM=\E[1;2H, kIC=\E[2;2~, kLFT=\E[1;2D, 2217 kNXT=\E[6;2~, kPRV=\E[5;2~, kRIT=\E[1;2C, ka1=\EOw, 2218 ka3=\EOy, kb2=\EOu, kbeg=\EOE, kbs=^?, kc1=\EOq, kc3=\EOs, 2219 kcbt=\E[Z, kcub1=\EOD, kcud1=\EOB, kcuf1=\EOC, kcuu1=\EOA, 2220 kdch1=\E[3~, kend=\EOF, kent=\EOM, kf1=\EOP, kf10=\E[21~, 2221 kf11=\E[23~, kf12=\E[24~, kf13=\E[1;2P, kf14=\E[1;2Q, 2222 kf15=\E[1;2R, kf16=\E[1;2S, kf17=\E[15;2~, kf18=\E[17;2~, 2223 kf19=\E[18;2~, kf2=\EOQ, kf20=\E[19;2~, kf21=\E[20;2~, 2224 kf22=\E[21;2~, kf23=\E[23;2~, kf24=\E[24;2~, 2225 kf25=\E[1;5P, kf26=\E[1;5Q, kf27=\E[1;5R, kf28=\E[1;5S, 2226 kf29=\E[15;5~, kf3=\EOR, kf30=\E[17;5~, kf31=\E[18;5~, 2227 kf32=\E[19;5~, kf33=\E[20;5~, kf34=\E[21;5~, 2228 kf35=\E[23;5~, kf36=\E[24;5~, kf37=\E[1;6P, kf38=\E[1;6Q, 2229 kf39=\E[1;6R, kf4=\EOS, kf40=\E[1;6S, kf41=\E[15;6~, 2230 kf42=\E[17;6~, kf43=\E[18;6~, kf44=\E[19;6~, 2231 kf45=\E[20;6~, kf46=\E[21;6~, kf47=\E[23;6~, 2232 kf48=\E[24;6~, kf49=\E[1;3P, kf5=\E[15~, kf50=\E[1;3Q, 2233 kf51=\E[1;3R, kf52=\E[1;3S, kf53=\E[15;3~, kf54=\E[17;3~, 2234 kf55=\E[18;3~, kf56=\E[19;3~, kf57=\E[20;3~, 2235 kf58=\E[21;3~, kf59=\E[23;3~, kf6=\E[17~, kf60=\E[24;3~, 2236 kf61=\E[1;4P, kf62=\E[1;4Q, kf63=\E[1;4R, kf7=\E[18~, 2237 kf8=\E[19~, kf9=\E[20~, khome=\EOH, kich1=\E[2~, 2238 kind=\E[1;2B, kmous=\E[<, knp=\E[6~, kpp=\E[5~, 2239 kri=\E[1;2A, mc0=\E[i, mc4=\E[4i, mc5=\E[5i, meml=\El, 2240 memu=\Em, mgc=\E[?69l, nel=\EE, oc=\E]104\007, 2241 op=\E[39;49m, rc=\E8, rep=%p1%c\E[%p2%{1}%-%db, 2242 rev=\E[7m, ri=\EM, rin=\E[%p1%dT, ritm=\E[23m, rmacs=\E(B, 2243 rmam=\E[?7l, rmcup=\E[?1049l\E[23;0;0t, rmir=\E[4l, 2244 rmkx=\E[?1l\E>, rmm=\E[?1034l, rmso=\E[27m, rmul=\E[24m, 2245 rs1=\Ec\E]104\007, rs2=\E[!p\E[?3;4l\E[4l\E>, sc=\E7, 2246 setab=\E[%?%p1%{8}%<%t4%p1%d%e%p1%{16}%<%t10%p1%{8}%-%d%e48;5;%p1%d%;m, 2247 setaf=\E[%?%p1%{8}%<%t3%p1%d%e%p1%{16}%<%t9%p1%{8}%-%d%e38;5;%p1%d%;m, 2248 sgr=%?%p9%t\E(0%e\E(B%;\E[0%?%p6%t;1%;%?%p5%t;2%;%?%p2%t;4%;%?%p1%p3%|%t;7%;%?%p4%t;5%;%?%p7%t;8%;m, 2249 sgr0=\E(B\E[m, sitm=\E[3m, smacs=\E(0, smam=\E[?7h, 2250 smcup=\E[?1049h\E[22;0;0t, smglp=\E[?69h\E[%i%p1%ds, 2251 smglr=\E[?69h\E[%i%p1%d;%p2%ds, 2252 smgrp=\E[?69h\E[%i;%p1%ds, smir=\E[4h, smkx=\E[?1h\E=, 2253 smm=\E[?1034h, smso=\E[7m, smul=\E[4m, tbc=\E[3g, 2254 u6=\E[%i%d;%dR, u7=\E[6n, u8=\E[?%[;0123456789]c, 2255 u9=\E[c, vpa=\E[%i%p1%dd, 2256</code></pre></div></div> 2257 2258<p>Thereâs so much junk in there. I wonder how much only applies to 2259non-ANSI hardware terminals, and therefore is irrelevant these days.</p> 2260 2261<p>For now, weâre only interested in three of these capabilities:</p> 2262<ul> 2263 <li><code class="language-plaintext highlighter-rouge">colors</code> is how many colors this terminal supports. The standard 2264values are 0, 8, 16, 256, and 0x1000000 (24-bit), though other 2265values exist.</li> 2266 <li><code class="language-plaintext highlighter-rouge">setaf</code> and <code class="language-plaintext highlighter-rouge">setab</code> set foreground and background colors, 2267respectively. I believe they stand for âSet ANSI Foregroundâ and 2268âSet ANSI Backgroundâ. Each takes a single argument, the color 2269number.</li> 2270</ul> 2271 2272<p>Those percent signs are a parameter arithmetic and substitution 2273language. Letâs decode <code class="language-plaintext highlighter-rouge">setaf</code> in particular:</p> 2274 2275<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>setaf=\E[%?%p1%{8}%<%t3%p1%d%e%p1%{16}%<%t9%p1%{8}%-%d%e38;5;%p1%d%;m 2276</code></pre></div></div> 2277 2278<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>print "\E[" 2279if p1 < 8 { 2280 print "3" p1 2281} else if p1 < 16 { 2282 print "9" (p1 - 8) 2283} else { 2284 print "38;5;" p1 2285}
2285 2286print "m" 2287</code></pre></div></div> 2288 2289<p>This is the <code class="language-plaintext highlighter-rouge">xterm-256color</code> terminfo description. It only knows how 2290to output the ANSI 30-37 SGR parameters, the non-standard 90-97 2291brights (from IBM AIX), or otherwise the 256-entry palette, using 2292ambiguous semicolon-delimited syntax.</p> 2293 2294<p>Letâs compare with <code class="language-plaintext highlighter-rouge">xterm-direct</code>, the terminfo entry that supports 2295RGB.</p> 2296 2297<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ infocmp xterm-256color xterm-direct 2298comparing xterm-256color to xterm-direct. 2299 comparing booleans. 2300 ccc: T:F. 2301 comparing numbers. 2302 colors: 256, 16777216. 2303 comparing strings. 2304 initc: '\E]4;%p1%d;rgb:%p2%{255}%*%{1000}%/%2.2X/%p3%{255}%*%{1000}%/%2.2X/%p4%{255}%*%{1000}%/%2.2X\E\\', NULL. 2305 oc: '\E]104\007', NULL. 2306 rs1: '\Ec\E]104\007', '\Ec'. 2307 setab: '\E[%?%p1%{8}%<%t4%p1%d%e%p1%{16}%<%t10%p1%{8}%-%d%e48;5;%p1%d%;m', '\E[%?%p1%{8}%<%t4%p1%d%e48:2::%p1%{65536}%/%d:%p1%{256}%/%{255}%&%d:%p1%{255}%&%d%;m'. 2308 setaf: '\E[%?%p1%{8}%<%t3%p1%d%e%p1%{16}%<%t9%p1%{8}%-%d%e38;5;%p1%d%;m', '\E[%?%p1%{8}%<%t3%p1%d%e38:2::%p1%{65536}%/%d:%p1%{256}%/%{255}%&%d:%p1%{255}%&%d%;m'. 2309</code></pre></div></div> 2310 2311<p>A few things are notable:</p> 2312 2313<ul> 2314 <li><code class="language-plaintext highlighter-rouge">xterm-direct</code> advertises 16.7 million colors, as expected.</li> 2315 <li><code class="language-plaintext highlighter-rouge">xterm-direct</code> unsets the <code class="language-plaintext highlighter-rouge">ccc</code> boolean, which indicates color 2316indices cannot have new RGB values assigned.</li> 2317 <li>Correspondingly, xterm-direct unsets <code class="language-plaintext highlighter-rouge">initc</code>, <code class="language-plaintext highlighter-rouge">oc</code>, and <code class="language-plaintext highlighter-rouge">rs1</code>, also 2318related to changing color values at runtime.</li> 2319 <li>And of course <code class="language-plaintext highlighter-rouge">setaf</code> and <code class="language-plaintext highlighter-rouge">setab</code> change. Weâll decode that next.</li> 2320</ul> 2321 2322<p>Hereâs where Terminfoâs limitations cause us trouble. Terminfo and 2323ncurses are tied at the hip. Their programming model is that there are 2324N palette entries, each of which has a default RGB value, and 2325terminals may support overriding any palette entryâs RGB value.</p> 2326 2327<p>The <code class="language-plaintext highlighter-rouge">-direct</code> terminals, however, are different. They represent 24-bit 2328colors by pretending there are 16.7 million palette entries, each of 2329which maps to the 8:8:8 RGB cube, but whose values cannot be changed.</p> 2330 2331<p>Now letâs look at the new <code class="language-plaintext highlighter-rouge">setaf</code>:</p> 2332 2333<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>print "\E[" 2334if p1 < 8 { 2335 print "3" p1 2336} else { 2337 print "38:2::" (p1 / 65536) ":" ((p1 / 256) & 255) ":" (p1 & 255) 2338} 2339print "m" 2340</code></pre></div></div> 2341 2342<p>Itâs not <em>quite</em> as simple as direct RGB. For compatibility with 2343programs that assume the meaning of <code class="language-plaintext highlighter-rouge">setaf</code>, this scheme steals the 2344darkest 7 blues, not including black, and uses them for compatibility 2345with the basic ANSI 8 colors. Otherwise, thereâs a risk of legacy 2346programs outputting barely-visible dark blues instead of the ANSI 2347colors they expect.</p> 2348 2349<p>One consequence is that the <code class="language-plaintext highlighter-rouge">-direct</code> schemes are incompatible with 2350the <code class="language-plaintext highlighter-rouge">-256color</code> schemes, so programs must be aware that 256 colors 2351means indexed and 16.7 million means direct, except that the darkest 7 2352blues are to be avoided.</p> 2353 2354<p>Fundamentally, terminfo has no notion of color space. So a program 2355that was written before terminfo even supported more colors than 256 2356might (validly!) assume the values of the first 8, 16, or even 256 2357palette entries.</p> 2358 2359<p>This explains an issue with the Rust crate 2360<a href="https://docs.rs/termwiz/latest/termwiz/">termwiz</a> that I <a href="https://github.com/wez/wezterm/issues/4528">recently 2361ran into</a> at work. A 2362<a href="https://sapling-scm.com/">program</a> expected to output colors in the 2363xterm-256color palette, but was actually generating various 2364illegibly-dark shades of blue. (Note: Despite the fact that the issue 2365is open as of this writing, @quark-zju landed a fix, so current 2366termwiz behaves reasonably.)</p> 2367 2368<p>This is a terminfo restriction, not a terminal restriction. As far as 2369I know, every terminal that supports 24-bit color also supports the 2370xterm 256-color palette and even dynamically changing their RGB 2371values. (You can even <a href="https://gist.github.com/chadaustin/7046bff2261b0f669d223a88ecad8282">animate the 2372palette</a> 2373like <a href="https://www.youtube.com/watch?v=aMcJ1Jvtef0">The Secret of Monkey Island 2374did</a>
2374!) While I appreciate 2375Thomas Dickeyâs dedication to accurately documenting history and 2376preserving compatibility, terminfo simply isnât great at accurate and 2377timely descriptions of todayâs vibrant ecosystem of terminal 2378emulators.</p> 2379 2380<p>Kovid Goyal, author of <a href="https://sw.kovidgoyal.net/kitty/">kitty</a>, 2381<a href="https://github.com/kovidgoyal/kitty/issues/4172#issuecomment-955190343">expresses his 2382frustration</a>:</p> 2383 2384<blockquote> 2385 <p>To summarize, one cannot have both 256 and direct color support in 2386one terminfo file.</p> 2387 2388 <p>Frustrated users of the ncurses library have only themselves to 2389blame, for choosing to use such a bad library.</p> 2390</blockquote> 2391 2392<p>A deeper, more accurate discussion of the challenges are documented in 2393<a href="https://github.com/kovidgoyal/kitty/issues/879">kitty issue #879</a>.</p> 2394 2395<p>In an ideal world, terminfo would have introduced a brand new 2396capability for 24-bit RGB, leaving the adjustable 256-color palette in 2397place.</p> 2398 2399<p>Modern programs should probably disregard most of terminfo and assume 2400that 16.7 million colors implies support for the rest of the color 2401capabilities. And maybe generate their own ANSI-compatible escape 2402sequences⦠except for the next wrinkle.</p> 2403 2404<h2 id="setting-term-semicolons-again">Setting TERM: Semicolons Again!</h2> 2405 2406<p>Gripes about terminfo aside, everyone uses it, so we do need to ensure 2407TERM is set correctly.</p> 2408 2409<p>While Iâd like to standardize on the colon-based SGR syntax, several 2410terminals I use only support semicolons:</p> 2411 2412<ul> 2413 <li><a href="https://learn.microsoft.com/en-us/windows/console/definitions#console-host">Conhost</a>, 2414Windowsâs built-in console.</li> 2415 <li><a href="https://github.com/mintty/mintty/wiki/Changelog#370-14-november-2023">Mintty</a> 2416<a href="https://github.com/mintty/mintty/wiki/Changelog#370-14-november-2023">claims to 2417work</a> 2418(and <a href="https://github.com/mintty/wsltty">wsltty</a> does), but for some 2419reason running <a href="https://gist.github.com/chadaustin/2d2c2cb4b71fd1d4163aa8115077624a">my colortest.rs 2420program</a> 2421from Cygwin only works with semicolon syntax, unless I pipe the 2422output through <code class="language-plaintext highlighter-rouge">cat</code> or a file. There must be some kind of magic 2423translation happening under the hood. I havenât debugged.</li> 2424 <li><a href="https://mosh.org/">Mosh</a> is aware, but hasnât <a href="https://github.com/mobile-shell/mosh/issues/951">added 2425support</a>.</li> 2426 <li><a href="https://www.chiark.greenend.org.uk/~sgtatham/putty/">PuTTY</a>.</li> 2427 <li>Ubuntu 22.04 LTS ships a version of Konsole that only supports 2428semicolons.</li> 2429</ul> 2430 2431<p>Terminfo entries are built from âbuilding blocksâ, marked with a plus. 2432<a href="https://invisible-island.net/ncurses/terminfo.src.html#tic-xterm_direct"><code class="language-plaintext highlighter-rouge">xterm+direct</code></a> 2433is the building block for the standard colon-delimited syntax. 2434<a href="https://invisible-island.net/ncurses/terminfo.src.html#tic-xterm_indirect"><code class="language-plaintext highlighter-rouge">xterm+indirect</code></a> 2435is the building block for legacy terminals that only support semicolon 2436syntax.</p> 2437 2438<p>Searching for <code class="language-plaintext highlighter-rouge">xterm+indirect</code> shows which terminfo entries might work 2439for me. <code class="language-plaintext highlighter-rouge">vscode-direct</code> looks the most accurate. I assume that, since 2440it targets a Microsoft terminal, itâs probably close enough in 2441functionality to Windows Terminal and Windows Console. I have not 2442audited all capabilities, but it seems to work.</p> 2443 2444<p>The next issue was that none of my servers had the <code class="language-plaintext highlighter-rouge">-direct</code> terminfo 2445entries installed! On most systems, the terminfo database comes from 2446the 2447<a href="https://packages.ubuntu.com/jammy/all/ncurses-base/filelist"><code class="language-plaintext highlighter-rouge">ncurses-base</code></a> 2448package, but you need 2449<a href="https://packages.ubuntu.com/jammy/all/ncurses-term/filelist"><code class="language-plaintext highlighter-rouge">ncurses-term</code></a> 2450for the extended set of terminals.</p> 2451 2452<p>At work, we can configure a default set of installed packages for your 2453hosts, but I have to install them manually on my unmanaged personal 2454home machines. Also, I was still running Ubuntu 18, so I had to 2455upgrade to a version that contained the <code class="language-plaintext highlighter-rouge">-direct</code> terminfo entries. 2456(Of course, two of my headless machines failed to boot after 2457upgrading, but thatâs a different story.)</p> 2458 2459<p><del>Unfortunately, there is no terminfo entry for the Windows console.</del> 2460Since I started writing this post, ncurses introduced a 2461<a href="https://invisible-island.net/ncurses/NEWS.html#index-t20231230">winconsole</a> 2462terminfo entry, but it neither supports 24-bit color nor is released 2463in any ncurses version.</p> 2464 2465<h2 id="configuring-emacs">Configuring Emacs</h2> 2466 2467<p>Emacs documents <a href="https://www.gnu.org/software/emacs/manual/html_node/efaq/Colors-on-a-TTY.html">how it detects truecolor 2468support</a>.</p> 2469 2470<p>I find it helpful to <code class="language-plaintext highlighter-rouge">M-x eval-expression</code> <code class="language-plaintext highlighter-rouge">(display-color-cells)</code> to 2471confirm whether Emacs sees 16.7 million colors.</p> 2472 2473<p>Emacs also documents the <code class="language-plaintext highlighter-rouge">-direct</code> mode terminfo limitation described 2474above:</p> 2475 2476<blockquote> 2477 <p>Terminals with âRGBâ capability treat pixels #000001 - #000007 as 2478indexed colors to maintain backward compatibility with applications 2479that are unaware of direct color mode. Therefore the seven darkest 2480blue shades may not be available. If this is a problem, you can 2481always use custom terminal definition with âsetb24â and âsetf24â.</p> 2482</blockquote> 2483 2484<p>Itâs worth noting that <code class="language-plaintext highlighter-rouge">RGB</code> is Emacsâs fallback capability. Emacs 2485looks for the <code class="language-plaintext highlighter-rouge">setf24</code> and <code class="language-plaintext highlighter-rouge">setb24</code> strings first, but no terminfo 2486entries on my machine contain those capabilities:</p> 2487 2488<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ for t in $(toe -a | cut -f1); do 2489 if (infocmp "$t" | grep 'setf24') > /dev/null; then 2490 echo "$t"; 2491 fi; 2492done 2493$ 2494</code></pre></div></div> 2495 2496<h2 id="nesting-terminals">Nesting Terminals</h2> 2497 2498<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>conhost.exe (WSL1) 2499+-------------------------+ 2500| mosh | 2501| +---------------------+ | 2502| | tmux | | 2503| | +-----------------+ | | 2504| | | emacs terminal | | | 2505| | | +-------------+ | | | 2506| | | | $ ls | | | | 2507| | | | foo bar baz | | | | 2508| | | +-------------+ | | | 2509| | +-----------------+ | | 2510| +---------------------+ | 2511+-------------------------+ 2512</code></pre></div></div> 2513 2514<p>Iâd never consciously considered this, but my typical workflow nests 2515multiple terminals.</p> 2516<ul> 2517 <li>I open a graphical terminal emulator on my local desktop, Windows, 2518Mac, or Linux.</li> 2519 <li>I mosh to a remote machine or VM.</li> 2520 <li>I start tmux.</li> 2521 <li>I might then use a terminal within Emacs or 2522<a href="https://asciinema.org/">Asciinema</a> or <a href="https://www.gnu.org/software/screen/">GNU 2523Screen</a>. 2524 <ul> 2525 <li>Yes, there are situations where itâs useful to have some screen 2526sessions running inside or outside of tmux.</li> 2527 </ul> 2528 </li> 2529</ul> 2530 2531<p>Each of those layers is its own implementation of the ANSI escape 2532sequence state machine. For 24-bit color to work, every single layer 2533has to understand and accurately translate the escape sequences from 2534the inner TERM valueâs terminfo to the outer terminfo.</p> 2535 2536<p>Therefore, you need recent-enough versions of all of this software. 2537Current LTS Ubuntus only ship with mosh 1.3, so I had to enable the 2538<a href="https://launchpad.net/~keithw/+archive/ubuntu/mosh-dev">mosh-dev 2539PPA</a>.</p> 2540 2541<p>TERM must be set correctly within each terminal: <code class="language-plaintext highlighter-rouge">tmux-direct</code> within 2542tmux, for example. There is no standard terminfo for <code class="language-plaintext highlighter-rouge">
2542mosh</code>, so you 2543have to pick something close enough.</p> 2544 2545<h3 id="graphical-terminal-emulators">Graphical Terminal Emulators</h3> 2546 2547<p>Most terminals either set TERM to a reasonable default or 2548allow you to override TERM.</p> 2549 2550<p>I use Konsole, but I think you could find a similar option in 2551whichever you use.</p> 2552 2553<figure> 2554<a href="/images/truecolor-terminal-emacs/konsole.png"><img src="/images/truecolor-terminal-emacs/konsole.png" alt="Konsole's TERM value selection" /></a> 2555<figcaption>Konsole's TERM value selection</figcaption> 2556</figure> 2557 2558<h3 id="ssh">ssh</h3> 2559 2560<p>Often, the first thing I do when opening a terminal is to <code class="language-plaintext highlighter-rouge">ssh</code> 2561somewhere else. Fortunately, this is easy, as long as the remote host 2562has the same terminfo record. <code class="language-plaintext highlighter-rouge">ssh</code> carries your TERM value into the 2563new shell.</p> 2564 2565<h3 id="tmux">tmux</h3> 2566 2567<p>But then you load <code class="language-plaintext highlighter-rouge">tmux</code> and TERM is set to <code class="language-plaintext highlighter-rouge">screen</code>! To fix this, 2568override <code class="language-plaintext highlighter-rouge">default-terminal</code> in your <code class="language-plaintext highlighter-rouge">~/.tmux.conf</code>:</p> 2569 2570<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>set -g default-terminal "tmux-direct" 2571</code></pre></div></div> 2572 2573<p>For extra credit, consider setting <code class="language-plaintext highlighter-rouge">tmux-direct</code> conditionally with 2574<code class="language-plaintext highlighter-rouge">%if</code> when the outer TERM supports 24-bit color, otherwise leaving the 2575default of <code class="language-plaintext highlighter-rouge">screen</code> or <code class="language-plaintext highlighter-rouge">tmux-256color</code>. And then let me know how you 2576did it. :P</p> 2577 2578<h3 id="mosh">mosh</h3> 2579 2580<p>While recent mosh does support 24-bit color, it <a href="https://github.com/mobile-shell/mosh/blob/1105d481bb9143dad43adf768f58da7b029fd39c/src/frontend/mosh-server.cc#L571">only advertises 8 or 2581256 2582colors</a>. 2583Thus, itâs up to you to set TERM appropriately.</p> 2584 2585<p>Mosh aims for xterm compatibility, but unfortunately only supports 2586semicolon syntax for SGR 38 and 48, so <code class="language-plaintext highlighter-rouge">TERM=xterm-direct</code> does not 2587work. So far, Iâve found that <code class="language-plaintext highlighter-rouge">vscode-direct</code> is the closest to 2588<code class="language-plaintext highlighter-rouge">xterm-direct</code>.</p> 2589 2590<p>There is no convenient âIâm running in moshâ variable, so I wrote a 2591<a href="https://gist.github.com/chadaustin/ee1a20e0522c10b65cb4006496d1fb7c"><code class="language-plaintext highlighter-rouge">detect-mosh.rs</code></a> 2592Rust script and called it from <code class="language-plaintext highlighter-rouge">.bashrc</code>:</p> 2593 2594<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">unamer</span><span class="o">=</span><span class="si">$(</span><span class="nb">uname</span> <span class="nt">-r</span><span class="si">)</span> 2595<span class="nv">unameo</span><span class="o">=</span><span class="si">$(</span><span class="nb">uname</span> <span class="nt">-o</span><span class="si">)</span> 2596<span class="k">if</span> <span class="o">[[</span> <span class="o">!</span> <span class="s2">"</span><span class="nv">$TMUX</span><span class="s2">"</span> <span class="o">]]</span><span class="p">;</span> <span class="k">then 2597 if</span> <span class="o">[[</span> <span class="s2">"</span><span class="nv">$unamer</span><span class="s2">"</span> <span class="o">==</span> <span class="k">*</span>Microsoft <span class="o">]]</span><span class="p">;</span> <span class="k">then</span> 2598 <span class="c"># WSL 1</span> 2599 <span class="nb">export </span><span class="nv">TERM</span><span class="o">=</span>vscode-direct 2600 <span class="k">elif</span> <span class="o">[[</span> <span class="s2">"</span><span class="nv">$unameo</span><span class="s2">"</span> <span class="o">==</span> Cygwin <span class="o">]]</span><span class="p">;</span> <span class="k">
2600then</span> 2601 <span class="c"># Eh, could just configure mintty to set mintty-direct.</span> 2602 <span class="nb">export </span><span class="nv">TERM</span><span class="o">=</span>vscode-direct 2603 <span class="k">elif </span>detect-mosh 2>/dev/null<span class="p">;</span> <span class="k">then</span> 2604 <span class="c"># This should be xterm-direct, but mosh does not understand SGR</span> 2605 <span class="c"># colon syntax.</span> 2606 <span class="nb">export </span><span class="nv">TERM</span><span class="o">=</span>vscode-direct 2607 <span class="k">fi 2608fi</span> 2609</code></pre></div></div> 2610 2611<p>It works by checking whether the shell process is a child of 2612<code class="language-plaintext highlighter-rouge">mosh-server</code>.</p> 2613 2614<p>The juryâs still out on whether itâs a good idea to compile Rust in 2615the critical path of login, especially into an underpowered host like 2616my Intel Atom NAS or a Raspberry Pi.</p> 2617 2618<h2 id="it-works">It Works!</h2> 2619 2620<p>Beautiful Emacs themes everywhere!</p> 2621 2622<figure> 2623<a href="/images/truecolor-terminal-emacs/finally.png"><img src="/images/truecolor-terminal-emacs/finally.png" alt="Emacs within tmux within mosh" /></a> 2624<figcaption>Emacs within tmux within mosh</figcaption> 2625</figure> 2626 2627<p>This was a ton of work, but I learned a lot, and, perhaps most 2628importantly, I now feel confident I could debug any kind of wonky 2629terminal behavior in the future.</p> 2630 2631<p>To recap:</p> 2632<ul> 2633 <li>Terminals donât agree on syntax and capabilities.</li> 2634 <li>Terminfo is how those capabilities are queried.</li> 2635 <li>Terminfo is often limited, sometimes inaccurate, and new terminfo 2636versions are released infrequently.</li> 2637</ul> 2638 2639<h2 id="whats-next">Whatâs Next?</h2> 2640 2641<p>If you were serious about writing software to take full advantage of 2642modern terminal capabilities, it would be time to break from terminfo.</p> 2643 2644<p>I imagine such a project would look like this:</p> 2645<ul> 2646 <li>Continue to use the TERM variable because itâs well-supported.</li> 2647 <li>Give programs knowledge of terminals independent of the age of the 2648operating system or distribution theyâre running on: 2649 <ul> 2650 <li>Programs would link with a frequently-updated (Rust?) library.</li> 2651 <li>Said library would contain a (modern!) terminfo database 2652representing, say, the last 10 years of terminal emulators, keyed 2653on (name, version). Notably, the library would not pretend to 2654support any hardware terminals, because they no longer exist. We 2655can safely forget about 2656<a href="https://www.gnu.org/software/termutils/manual/termcap-1.3/html_mono/termcap.html#SEC7">padding</a>, 2657for example.</li> 2658 </ul> 2659 </li> 2660 <li>Continue to support the terminfo file format and OS-provided 2661terminfo files on disk, with some protocol for determining which 2662information is most-up-to-date.</li> 2663 <li>Allow an opt-in TERMVERSION to differentiate between the 2664capabilities of, for example, 2022âs Konsole and 2023âs Konsole.</li> 2665 <li>Allow describing modern terminal capabilities (like 24-bit color, 2666256-color palette animation, <a href="https://github.com/Alhadis/OSC8-Adoption/">URL 2667links</a>, <a href="https://sw.kovidgoyal.net/kitty/graphics-protocol/">Kittyâs graphics 2668protocol</a>) in an 2669accurate, unambiguous format, independent of the timeline of new 2670ncurses releases.</li> 2671 <li>Backport modern terminal descriptions to legacy programs by 2672providing a program to be run by <code class="language-plaintext highlighter-rouge">.bashrc</code> that: 2673 <ul> 2674 <li>Uses TERM and TERMVERSION to generate a binary terminfo file in 2675<code class="language-plaintext highlighter-rouge">$HOME/.terminfo/</code>, which ncurses knows how to discover.</li> 2676 <li>Generates unambiguous 24-bit color capabilities like <code class="language-plaintext highlighter-rouge">RGB</code>, 2677<code class="language-plaintext highlighter-rouge">setf24</code>, and <code class="language-plaintext highlighter-rouge">setb24</code>, despite the fact that getting them added 2678to terminfo has been politically untenable.</li> 2679 <li>Otherwise, assumes RGB-unaware programs will assume the 256-color 2680palette, and leaves <code class="language-plaintext highlighter-rouge">colors#0x100</code>, <code class="language-plaintext highlighter-rouge">initc</code>, <code class="language-plaintext highlighter-rouge">oc</code> in place. 2681Palette animation is a useful, widely-supported feature.</li> 2682 </ul> 2683 </li> 2684</ul> 2685 2686<p>Let me know if youâre interested in such a project!</p> 2687 2688 </li></ul> 2689 </div><div class="home-sidebar"><div class="sidebar-section"> 2690 <form class="search" action="https://duckduckgo.com" method="get"> 2691 <input class="search-text" type="text" name="q" placeholder="Search..."></input> 2692 <input type="hidden" name="sites" value="chadaustin.me"></input> 2693 <input type="image" alt="Search" src="/assets/search.svg"></input> 2694 </form> 2695 </div> 2696 <div class="sidebar-section sidebar-subscribe-section"> 2697 <div class="subscribe-options"> 2698 <div class="subscribe twitter"> 2699 <a rel="noopener noreferrer" target="_blank" href="https://mastodon.gamedev.place/@chadaustin"><img class="twitter-icon" alt="Mastodon icon" src="/assets/mastodon-icon.svg" width='24' height='24' /> 2700 follow on Mastodon</a> 2701 </div> 2702 <div class="subscribe feedly"> 2703 <a rel="noopener noreferrer" target="_blank" href='https://feedly.com/i/subscription/feed%2Fhttp%3A%2F%2Faegisknight.org%2Ffeed' target='blank'><img class="feedly-icon" alt='Feedly icon' src='/assets/feedly-icon.svg' width='27' height='24' /> 2704 follow with Feedly</a> 2705 </div> 2706 <div class="subscribe atom"> 2707 <a rel="noopener noreferrer" href="http://aegisknight.org/feed">
2707<img class="rss-icon" alt="Atom feed icon" src="/assets/feed-icon.svg" width='24' height='24' /> 2708 RSS for the rest</a> 2709 </div> 2710</div> 2711 2712 </div> 2713 <div class="sidebar-section"> 2714 <img alt="Chad" class="portrait" src="/images/chad.png" /> 2715 <p>Hi, I'm Chad. I work at Facebook. Before that, 2716 <a href="/tag/dropbox">Dropbox</a> and <a href="/tag/imvu">IMVU</a>. 2717 I care about usability and open source and products that work well.</p> 2718 <p>I've worked in many areas: games, graphics, mobile apps, audio, optimization, build systems, Haskell, concurrency, and compilers.</p> 2719 <p><a href="/about">The longer story...</a></p> 2720 </div> 2721 <div class="sidebar-section recent-posts"> 2722 <p class="sidebar-section-title">Recent Posts</p> 2723 <ul><li><a class="recent-post-link" href="/2025/03/snes-classic-partial-repair/"> 2724 (Partially) Repairing a Super NES Classic 2725 </a></li><li><a class="recent-post-link" href="/2024/10/intrusive-linked-list-in-rust/"> 2726 Unsafe Rust Is Harder Than C 2727 </a></li><li><a class="recent-post-link" href="/2024/02/windows-terminal-latency/"> 2728 Terminal Latency on Windows 2729 </a></li><li><a class="recent-post-link" href="/2024/02/tmux-config/"> 2730 My Minimal tmux Config 2731 </a></li><li><a class="recent-post-link" href="/2024/01/truecolor-terminal-emacs/"> 2732 I Just Wanted Emacs to Look Nice â Using 24-Bit Color in Terminals 2733 </a></li></ul> 2734 </div> 2735 <div class="sidebar-section"> 2736 <p class="sidebar-section-title"><a href="/bestof">Popular posts...</a></p> 2737 </div> 2738 <div class="sidebar-section"> 2739 <p class="sidebar-section-title">Tags</p><div class="tag-cloud"><a class="tag-link tag-count-3" href="/tag/agile">agile</a> 2740 <a class="tag-link tag-count-2" href="/tag/atom">atom</a> 2741 <a class="tag-link tag-count-1" href="/2010/07/book-review-e-myth-revisited/">book</a> 2742 <a class="tag-link tag-count-5" href="/tag/c++">c++</a> 2743 <a class="tag-link tag-count-1" href="/2006/09/download-imvus-cal3d-modifications/">cal3d</a> 2744 <a class="tag-link tag-count-1" href="/2019/11/two-years-at-dropbox/">career</a> 2745 <a class="tag-link tag-count-2" href="/tag/concurrency">concurrency</a> 2746 <a class="tag-link tag-count-4" href="/tag/crashes">crashes</a> 2747 <a class="tag-link tag-count-3" href="/tag/crux">crux</a> 2748 <a class="tag-link tag-count-3" href="/tag/design">design</a> 2749 <a class="tag-link tag-count-2" href="/tag/dropbox">dropbox</a> 2750 <a class="tag-link tag-count-2" href="/tag/electronics">electronics</a> 2751 <a class="tag-link tag-count-2" href="/tag/embind">embind</a> 2752 <a class="tag-link tag-count-4" href="/tag/emscripten">emscripten</a> 2753 <a class="tag-link tag-count-2" href="/tag/entrepreneurship">entrepreneurship</a> 2754 <a class="tag-link tag-count-1" href="/2018/02/timestamps/">facebook</a> 2755 <a class="tag-link tag-count-3" href="/tag/family">family</a> 2756 <a class="tag-link tag-count-3" href="/tag/flash">flash</a> 2757 <a class="tag-link tag-count-3" href="/tag/food">food</a> 2758 <a class="tag-link tag-count-3" href="/tag/functional">functional</a> 2759 <a class="tag-link tag-count-1" href="/2014/10/the-gamepad-api/">gamepad</a> 2760 <a class="tag-link tag-count-4" href="/tag/games">games</a> 2761 <a class="tag-link tag-count-1" href="/2010/01/comparing-ff3-to-ffantasy-iv/">games-design</a> 2762 <a class="tag-link tag-count-3" href="/tag/gdc">gdc</a> 2763 <a class="tag-link tag-count-1" href="/2010/03/your-version-control-and-build-systems-dont-scale-introducing-ibb/">git</a> 2764 <a class="tag-link tag-count-3" href="/tag/graphics">graphics</a> 2765 <a class="tag-link tag-count-4" href="/tag/hackernews">hackernews</a> 2766 <a class="tag-link tag-count-4" href="/tag/haskell">haskell</a> 2767 <a class="tag-link tag-count-2" href="/tag/http">http</a> 2768 <a class="tag-link tag-count-2" href="/tag/ibb">ibb</a> 2769 <a class="tag-link tag-count-5" href="/tag/imvu">imvu</a> 2770 <a class="tag-link tag-count-2" href="/tag/iphone">iphone</a> 2771 <a class="tag-link tag-count-4" href="/tag/javascript">javascript</a> 2772 <a class="tag-link tag-count-4" href="/tag/json">json</a>
2773 <a class="tag-link tag-count-1" href="/2021/02/wired-sculpt/">keyboard</a> 2774 <a class="tag-link tag-count-3" href="/tag/life">life</a> 2775 <a class="tag-link tag-count-2" href="/tag/linux">linux</a> 2776 2777 <a class="tag-link tag-count-3" href="/tag/memory">memory</a> 2778 <a class="tag-link tag-count-2" href="/tag/mozilla">mozilla</a> 2779 <a class="tag-link tag-count-3" href="/tag/nativeclient">nativeclient</a> 2780 2781 <a class="tag-link tag-count-3" href="/tag/opengl">opengl</a> 2782 <a class="tag-link tag-count-4" href="/tag/opensource">opensource</a> 2783 <a class="tag-link tag-count-5" href="/tag/performance">performance</a> 2784 <a class="tag-link tag-count-1" href="/2006/10/writing-solid-php/">php</a> 2785 <a class="tag-link tag-count-1" href="/2014/12/my-political-views/">politics</a> 2786 <a class="tag-link tag-count-2" href="/tag/priority">priority</a> 2787 <a class="tag-link tag-count-4" href="/tag/python">python</a> 2788 <a class="tag-link tag-count-4" href="/tag/reddit">reddit</a> 2789 <a class="tag-link tag-count-2" href="/tag/rust">rust</a> 2790 <a class="tag-link tag-count-3" href="/tag/sajson">sajson</a> 2791 <a class="tag-link tag-count-3" href="/tag/scons">scons</a> 2792 <a class="tag-link tag-count-2" href="/tag/seh">seh</a> 2793 <a class="tag-link tag-count-2" href="/tag/slides">slides</a> 2794 <a class="tag-link tag-count-1" href="/2015/01/code-reviews-follow-the-data/">software-engineering</a> 2795 <a class="tag-link tag-count-1" href="/2014/10/http2-request-priorities-a-summary/">spdy</a> 2796 <a class="tag-link tag-count-2" href="/tag/tasks">tasks</a> 2797 <a class="tag-link tag-count-2" href="/tag/tdd">tdd</a> 2798 <a class="tag-link tag-count-3" href="/tag/terminal">terminal</a> 2799 <a class="tag-link tag-count-1" href="/2015/04/the-long-term-problem-with-dynamically-typed-languages/">types</a> 2800 <a class="tag-link tag-count-3" href="/tag/webapi">webapi</a> 2801 <a class="tag-link tag-count-1" href="/2014/09/web-platform-limitations-part-2-web-audio-api-is-a-mess/">webaudio</a> 2802 <a class="tag-link tag-count-1" href="/2014/09/optimizing-webgl-shaders-by-reading-d3d-shader-assembly/">webgl</a> 2803 <a class="tag-link tag-count-4" href="/tag/x86">x86</a> 2804 </div> 2805</div> 2806</div> 2807</div> 2808 2809 </div> 2810 </main><footer> 2811 <div class="wrapper"><p class="copyright">© 2812 2025 Chad Austin. 2813 To publish a translation, please <a href="/contact">contact me</a>. 2814 </p></div> 2815</footer> 2816</body> 2817 2818</html>
Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.