1const e={title:"A GPU Scheduler for Local LLMs: What Broke on Real Hardware",date:"2026-10-01",description:"How GridCore shares one 16 GB GPU between chat, agents and background jobs: design rules, VRAM estimates and the bugs only real hardware showed."},t=`<p>Several things want my GPU at the same time: the chat I'm typing into, a coding agent, an indexer working through incoming documents, and a model that reads my screenshots. There is one card â an RTX 4060 Ti with 16 GB. Give each app its own <code>llama-server</code> and they know nothing about each other: while memory lasts, they split the GPU evenly, and when it runs out, something crashes. Ollama can unload models, but on a timer, with no idea that a chat matters more than indexing.</p> 2<p>So in June I started writing GridCore, a scheduler that sits between applications and the GPU. <a href="https://www.youtube.com/watch?v=Mu3xzCVoHXc">The video</a> shows what it does. This note is about how it's built â and what only broke once it met real hardware.</p> 3<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/Mu3xzCVoHXc" title="Run Multiple Local LLMs on One GPU Without Slowdowns" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe><h2>A control plane, not another inference engine</h2> 4<p>GridCore doesn't run inference itself. llama.cpp does that, one <code>llama-server</code> process per model. GridCore starts and stops those processes, accepts requests over the OpenAI API, and decides who gets the GPU right now.</p> 5<figure class="note-figure"><svg viewBox="0 0 680 572" role="img" aria-labelledby="gc-arch-title gc-arch-desc" xmlns="http://www.w3.org/2000/svg"> 6<title id="gc-arch-title">How GridCore sits between applications and the GPU</title> 7<desc id="gc-arch-desc">Chat, a coding agent, a document indexer and a screenshot analyzer talk to GridCore over the OpenAI API. GridCore holds the priority queues, VRAM accounting, model residency and admission control, and runs one llama-server process per model: gemma4-12b (hot, 8.3 GB), gemma4-e4b (cold, 4.0 GB), gemma4-e2b (cold, 2.8 GB) and nomic-embed (pinned, 0.4 GB), together 15.5 of the 16 GB on an RTX 4060 Ti.</desc> 8<g font-family="'IBM Plex Sans', sans-serif"> 9<text x="340.0" y="22" font-size="11.5" font-weight="600" fill="#6b7785" text-anchor="middle" letter-spacing="1">APPS · AGENTS · BACKGROUND SERVICES</text> 10<rect x="30" y="34" width="146" height="44" rx="8" fill="#fff" stroke="#d3dce5"/> 11<text x="103.0" y="61" font-size="14.5" font-weight="600" fill="#333" text-anchor="middle">Chat</text> 12<rect x="188" y="34" width="146" height="44" rx="8" fill="#fff" stroke="#d3dce5"/> 13<text x="261.0" y="61" font-size="14.5" font-weight="600" fill="#333" text-anchor="middle">Coding agent</text> 14<rect x="346" y="34" width="146" height="44" rx="8" fill="#fff" stroke="#d3dce5"/> 15<text x="419.0" y="61" font-size="14.5" font-weight="600" fill="#333" text-anchor="middle">Document indexer</text> 16<rect x="504" y="34" width="146" height="44" rx="8" fill="#fff" stroke="#d3dce5"/> 17<text x="577.0" y="61" font-size="14.5" font-weight="600" fill="#333" text-anchor="middle">Screenshots</text> 18<line x1="103.0" y1="78" x2="103.0" y2="156" stroke="#8a9aab" stroke-width="1.5"/> 19<path d="M98.0 154L103.0 162L108.0 154z" fill="#8a9aab"/> 20<line x1="261.0" y1="78" x2="261.0" y2="156" stroke="#8a9aab" stroke-width="1.5"/> 21<path d="M256.0 154L261.0 162L266.0 154z" fill="#8a9aab"/> 22<line x1="419.0" y1="78" x2="419.0" y2="156" stroke="#8a9aab" stroke-width="1.5"/> 23<path d="M414.0 154L419.0 162L424.0 154z" fill="#8a9aab"/> 24<line x1="577.0" y1="78" x2="577.0" y2="156" stroke="#8a9aab" stroke-width="1.5"/> 25<path d="M572.0 154L577.0 162L582.0 154z" fill="#8a9aab"/> 26<rect x="170" y="105" width="340" height="30" rx="15" fill="#fff" stroke="#d3dce5"/> 27<text x="340.0" y="125" font-size="13" fill="#333" text-anchor="middle"><tspan font-weight="600">OpenAI API</tspan> + optional X-GridCore-* headers</text> 28<rect x="20" y="162" width="640" height="200" rx="12" fill="#264d6d"/> 29<text x="44" y="197" font-size="21" font-weight="700" fill="#fff" text-anchor="start">GridCore</text> 30<text x="636" y="196" font-size="13" font-weight="400" fill="#cfdce6" text-anchor="end">the scheduler that owns the GPU</text> 31<rect x="44" y="214" width="288" height="56" rx="8" fill="#fff" fill-opacity="0.09" stroke="#fff" stroke-opacity="0.2"/> 32<text x="60" y="238" font-size="15" font-weight="600" fill="#fff" text-anchor="start">Priority queues</text> 33<text x="60" y="257" font-size="12.5" font-weight="400" fill="#cfdce6" text-anchor="start">interactive ⺠background ⺠batch</text> 34<rect x="348" y="214" width="288" height="56" rx="8" fill="#fff" fill-opacity="0.09" stroke="#fff" stroke-opacity="0.2"/> 35<text x="364" y="238" font-size="15" font-weight="600" fill="#fff" text-anchor="start">VRAM accounting</text> 36<text x="364" y="257" font-size="12.5" font-weight="400" fill="#cfdce6" text-anchor="start">estimated from GGUF, then measured</text> 37<rect x="44" y="282" width="288" height="56" rx="8" fill="#fff" fill-opacity="0.09" stroke="#fff" stroke-opacity="0.2"/> 38<text x="60" y="306" font-size="15" font-weight="600" fill="#fff" text-anchor="start">Model residency</text> 39<text x="60" y="325" font-size="12.5" font-weight="400" fill="#cfdce6" text-anchor="start">pinned · hot · cold, LRU eviction</text> 40<rect x="348" y="282" width="288" height="56" rx="8" fill="#fff" fill-opacity="0.09" stroke="#fff" stroke-opacity="0.2"/> 41<text x="364" y="306" font-size="15" font-weight="600" fill="#fff" text-anchor="start">Admission</text> 42<text x="364" y="325" font-size="12.5" font-weight="400" fill="#cfdce6" text-anchor="start">dispatch, load, evict or wait</text> 43<line x1="103.0" y1="362" x2="103.0" y2="396" stroke="#8a9aab" stroke-width="1.5"/> 44<path d="M98.0 394L103.0 402L108.0 394z" fill="#8a9aab"/>
45<line x1="261.0" y1="362" x2="261.0" y2="396" stroke="#8a9aab" stroke-width="1.5"/> 46<path d="M256.0 394L261.0 402L266.0 394z" fill="#8a9aab"/> 47<line x1="419.0" y1="362" x2="419.0" y2="396" stroke="#8a9aab" stroke-width="1.5"/> 48<path d="M414.0 394L419.0 402L424.0 394z" fill="#8a9aab"/> 49<line x1="577.0" y1="362" x2="577.0" y2="396" stroke="#8a9aab" stroke-width="1.5"/> 50<path d="M572.0 394L577.0 402L582.0 394z" fill="#8a9aab"/> 51<rect x="30" y="402" width="146" height="56" rx="8" fill="#fff" stroke="#d3dce5"/> 52<circle cx="48" cy="424" r="5" fill="#d9822b"/> 53<text x="60" y="429" font-size="13.5" font-weight="600" fill="#333" text-anchor="start" font-family="'IBM Plex Mono', monospace">gemma4-12b</text> 54<text x="60" y="447" font-size="12" font-weight="400" fill="#6b7785" text-anchor="start">hot · 8.3 GB</text> 55<rect x="188" y="402" width="146" height="56" rx="8" fill="#fff" stroke="#d3dce5"/> 56<circle cx="206" cy="424" r="5" fill="#5b8fbd"/> 57<text x="218" y="429" font-size="13.5" font-weight="600" fill="#333" text-anchor="start" font-family="'IBM Plex Mono', monospace">gemma4-e4b</text> 58<text x="218" y="447" font-size="12" font-weight="400" fill="#6b7785" text-anchor="start">cold · 4.0 GB</text> 59<rect x="346" y="402" width="146" height="56" rx="8" fill="#fff" stroke="#d3dce5"/> 60<circle cx="364" cy="424" r="5" fill="#9dbfdc"/> 61<text x="376" y="429" font-size="13.5" font-weight="600" fill="#333" text-anchor="start" font-family="'IBM Plex Mono', monospace">gemma4-e2b</text> 62<text x="376" y="447" font-size="12" font-weight="400" fill="#6b7785" text-anchor="start">cold · 2.8 GB</text> 63<rect x="504" y="402" width="146" height="56" rx="8" fill="#fff" stroke="#d3dce5"/> 64<circle cx="522" cy="424" r="5" fill="#2e9d63"/> 65<text x="534" y="429" font-size="13.5" font-weight="600" fill="#333" text-anchor="start" font-family="'IBM Plex Mono', monospace">nomic-embed</text> 66<text x="534" y="447" font-size="12" font-weight="400" fill="#6b7785" text-anchor="start">pinned · 0.4 GB</text> 67<text x="340.0" y="482" font-size="12" font-weight="400" fill="#6b7785" text-anchor="middle">one llama-server process per model · llama.cpp does the inference</text> 68<text x="30" y="512" font-size="13" font-weight="600" fill="#333" text-anchor="start">GPU · RTX 4060 Ti, 16 GB</text> 69<text x="650" y="512" font-size="12.5" font-weight="400" fill="#6b7785" text-anchor="end">15.5 of 16 GB in use</text> 70<clipPath id="gc-arch-bar"><rect x="30" y="522" width="620" height="28" rx="6"/></clipPath> 71<rect x="30" y="522" width="620" height="28" rx="6" fill="#e7ecf1"/> 72<g clip-path="url(#gc-arch-bar)"> 73<rect x="30.0" y="522" width="319.6" height="28" fill="#d9822b"/> 74<rect x="351.6" y="522" width="153.0" height="28" fill="#5b8fbd"/> 75<rect x="506.6" y="522" width="106.5" height="28" fill="#9dbfdc"/> 76<rect x="615.1" y="522" width="13.5" height="28" fill="#2e9d63"/> 77</g> 78<text x="190.8" y="541" font-size="12" font-weight="600" fill="#fff" text-anchor="middle">8.3 GB</text> 79<text x="429.1" y="541" font-size="12" font-weight="600" fill="#fff" text-anchor="middle">4.0 GB</text> 80<text x="560.9" y="541" font-size="12" font-weight="600" fill="#1d3d57" text-anchor="middle">2.8 GB</text> 81</g></svg></figure><p>For applications, only the address changes. The endpoints are the same, and aliases in the config let an existing client keep sending <code>"model": "gpt-4o"</code> without knowing that a local Gemma is behind it. The one extra thing an app can tell GridCore is its class â <code>interactive</code>, <code>background</code> or <code>batch</code> â through a header, a field in the request body (any OpenAI SDK can send it via <code>extra_body</code>), or a suffix on the model name:</p> 82<pre class="shiki github-light" style="color:#24292e" tabindex="0"><code><span class="line"><span style="color:#6F42C1">curl</span><span style="color:#032F62"> localhost:8080/v1/embeddings</span><span style="color:#005CC5"> \\</span></span> 83<span class="line"><span style="color:#005CC5"> -H</span><span style="color:#032F62"> 'X-GridCore-Class: background'</span><span style="color:#005CC5"> \\</span></span> 84<span class="line"><span style="color:#005CC5"> -d</span><span style="color:#032F62"> '{"model": "nomic-embed", "input": ["...", "..."]}'</span></span></code></pre><p>A request without a class is treated as interactive.</p> 85<p>The first question I get: why bother, when <code>llama-server</code> has had router mode since December 2025? Router mode also runs one process per model and also loads and unloads them, but it evicts the least recently used model once their number exceeds <code>--models-max</code>. It doesn't account for VRAM, it has no priorities, and its child processes are hidden inside it â so there is no way to measure how much memory each model actually takes. And those numbers are what every scheduling decision is built on.
85</p> 86<p>GridCore is written in Go: a single static binary with no cgo, two dependencies (<code>yaml.v3</code> and the Prometheus client), running as a systemd user service. The daemon is about 7,500 lines of code, with 120 tests.</p> 87<h2>A few hard rules instead of one clever algorithm</h2> 88<p>I wanted the scheduler's behavior to be predictable from the config alone. So at its core are a handful of simple rules:</p> 89<ul> 90<li><strong>Strict class order.</strong> Interactive always goes first; within a class it's FIFO. No aging, where a job that has waited long enough gradually climbs in priority: sooner or later that would let a batch job overtake a chat.</li> 91<li><strong>Preemption only between steps.</strong> A chat response is one indivisible step: GridCore never interrupts a generation halfway through. Background work, by contrast, is cut into small steps: a large <code>/v1/embeddings</code> request is split into chunks of 32 inputs, each scheduled separately, and the client still gets a single assembled response. Between steps, the GPU can go to the chat.</li> 92<li><strong>Interactive mode.</strong> While an interactive request is running, and for two seconds after it, no new background steps start. The two seconds keep the card from going to background work in the pause between two messages of the same conversation.</li> 93<li><strong>Residency tiers.</strong> A model used interactively in the last five minutes is <em>hot</em>: background work can't evict it. The rest are <em>cold</em> and get evicted least-recently-used first. <em>Pinned</em> models â for me, a 0.4 GB embedder â are never evicted.</li> 94<li><strong>Invariant number one.</strong> A process with requests in flight is never stopped. The scheduler first stops giving it work, waits for the running requests to finish, and only then unloads it.</li> 95</ul> 96<p>All state belongs to a single goroutine: HTTP handlers only enqueue a job and wait for permission to run. A model that starts loading reserves its memory immediately, so two concurrent decisions can't claim the same space twice.</p> 97<p>Here is what that looks like in the event log when a user starts chatting 300 ms after background indexing began:</p> 98<pre class="shiki github-light" style="color:#24292e" tabindex="0"><code><span class="line"><span>11:20:27.194 enqueue background embedding nomic-embed steps=32</span></span> 99<span class="line"><span>11:20:27.194 dispatch background step 1/32 -> nomic-embed</span></span> 100<span class="line"><span>11:20:27.500 enqueue interactive chat gemma4-12b steps=1</span></span> 101<span class="line"><span>11:20:27.500 dispatch interactive step 1/1 -> gemma4-12b</span></span> 102<span class="line"><span>11:20:27.500 mode gpu interactive</span></span> 103<span class="line"><span>11:20:28.731 complete interactive gemma4-12b</span></span> 104<span class="line"><span>11:20:30.833 mode gpu idle</span></span> 105<span class="line"><span>11:20:31.032 dispatch background step 32/32 -> nomic-embed</span></span></code></pre><p>The chat got the GPU in the same millisecond, with no time in the queue. The background job waited for the answer to finish and the two-second window to pass, then picked up where it had left off.</p> 106<h2>Count first, load second</h2> 107<p>To decide whether a model fits, you need to know its VRAM footprint before loading it. The first version estimated it from the file size and was off by anywhere from â8% to +94%. The worst case was Gemma 4 E2B: an estimate of 5678 MB against an actual 2924. That model has a huge per-layer embedding table, which llama.cpp keeps in system RAM rather than on the card, so the file size tells you almost nothing.</p> 108<p>Now GridCore reads the GGUF header and counts tensor by tensor: the weights, minus the embedding tables that stay in RAM; the KV cache, layer by layer, accounting for sliding-window layers, Gemma's shared-KV layers and hybrid SSM architectures; plus the vision projector and runtime buffers. On the models I tested, the error is between â1% and +10%. After the first load, the estimate is replaced by the memory the process actually took on the card.</p> 109<p>Two mistakes worth knowing about if you do this math yourself:</p> 110<ul> 111<li><code>--ctx-size</code> in <code>llama-server</code> is the total context pool shared by all slots, not a per-slot size. As long as I multiplied it by the number of slots, the Qwen3-14B estimate came out at 12.7 GB against a real 9.7.</li> 112<li>Placement flags change everything. Ornith-1.5-35B-A3B with <code>--cpu-moe</code> keeps 18.6 GB of experts in system RAM and takes 2.2 GB on the card â and still generates 46 tokens per second, as fast as a 9B model sitting entirely on the GPU. Without accounting for the flag, the estimate was 21 GB and the scheduler refused to load the model at all. Now it understands <code>-ngl</code>, <code>--cpu-moe</code>, <code>--n-cpu-moe</code> and <code>-ot</code>: 2229 MB estimated against 2244 measured.</li> 113</ul> 114<p>This also explains a decision that looks odd at first glance: GridCore doesn't use the automatic placement fitting that llama.cpp has added. Auto-fit adapts to whatever memory is free at the moment of loading, while the scheduler needs a predictable footprint before it. So placement is set explicitly in each model's config.</p> 115<p>For the same reason, GridCore has no KV cache saturation to fight and never frees llama.cpp's slots by hand. <code>llama-server</code> allocates the entire KV cache once, when the model loads: <code>--ctx-size</code> is the total pool, <code>--parallel</code> splits it evenly into slots, and that amount doesn't grow at runtime. GridCore doesn't touch the slots either: it tracks how many are busy itself and never sends a model more requests than it has slots. Everything else waits in GridCore'
115s priority queue rather than in <code>llama-server</code>'s internal queue â which is exactly what lets a chat overtake background work. If a request is longer than one slot's context, <code>llama-server</code> returns <code>exceed_context_size_error</code>, and GridCore passes that error back to the client as is, without truncating anything. The price of this approach is that KV cache memory is reserved for the worst case, even while slots sit idle. In exchange, you know in advance exactly how much memory a model will take.</p> 116<h2>What only broke on real hardware</h2> 117<p>On the simulated test GPU and simulated runtime, every scenario test passed. Then I ran a five-minute load test on the real card â 6 chat clients, 8 background, 4 batch and 2 embedding clients â and it found what the tests couldn't see.</p> 118<ul> 119<li><strong>Complete background starvation.</strong> Six chat clients never once let the GPU leave interactive mode, and in five minutes the background jobs completed zero requests. Under constant load, "background waits while chat is quiet" means "background never runs". Now, if the head of the background queue hasn't made progress for 3 seconds, one background step may run alongside the chat. The first version of that rule didn't work either: the oldest starved job was waiting for a model that couldn't fit next to two hot models, and it blocked embedding jobs whose model was already loaded. Now the scheduler walks the starved jobs from oldest to newest and runs the first one that actually can run.</li> 120<li><strong>Excess evictions.</strong> To free 200 MB, the scheduler unloaded an 8.5 GB model, then another 6.8 GB one. The first victim was still finishing its requests and its memory still counted as used â so the scheduler picked a second victim. Now the memory of models already marked for eviction counts as being freed.</li> 121<li><strong>Phantom memory.</strong> <code>nvidia-smi</code> reports total used memory and the per-process list in two separate calls. If a <code>llama-server</code> exits between them, for one poll its memory looks like ~11 GB taken by some outside process. The scheduler concluded there was no room and, in the same millisecond, unloaded a second, perfectly healthy model. Now the estimate of outside memory is frozen while a process is shutting down and for two polls after.</li> 122<li><strong>502s out of nowhere.</strong> Three errors in five minutes, and nothing in the <code>llama-server</code> logs. The HTTP server inside llama.cpp closes a connection after five requests, and Go kept picking already-closed connections from its pool. One connection per request over loopback turned out to be cheaper than retries.</li> 123</ul> 124<p>Each of these bugs became a regression test, and the simulated GPU learned things it couldn't do before â such as holding on to the memory of a process that is already gone from the list.</p> 125<h2>The numbers</h2> 126<p>All on the same RTX 4060 Ti 16 GB. During the load test, four models lived on the card at once â Gemma 4 12B for chat, Gemma 4 E4B and E2B for background and batch work, and the embedder â using 15.5 of 16 GB, with not a single unnecessary load and not a single error in five minutes:</p> 127<table> 128<thead> 129<tr> 130<th>Class</th> 131<th>Requests</th> 132<th>p50</th> 133<th>p95</th> 134</tr> 135</thead> 136<tbody><tr> 137<td>interactive</td> 138<td>366</td> 139<td>3.4 s</td> 140<td>4.1 s</td> 141</tr> 142<tr> 143<td>background</td> 144<td>194</td> 145<td>11.8 s</td> 146<td>13.3 s</td> 147</tr> 148<tr> 149<td>batch</td> 150<td>96</td> 151<td>11.8 s</td> 152<td>13.4 s</td> 153</tr> 154<tr> 155<td>embeddings</td> 156<td>21</td> 157<td>25 s</td> 158<td>34 s</td> 159</tr> 160</tbody></table> 161<p>The honest cost: under mixed load, chat p50 went from 1.9 to 3.4 seconds. Part of the difference is six clients sharing two model slots, part is that one background step running next to the chat. If chat needs full exclusivity, <code>background_max_starvation: 0s</code> restores it â but then, under constant chat, background work doesn't move at all.</p> 162<h2>What v0.1 doesn't do</h2> 163<p>Deliberately left out: interrupting a chat mid-generation, multiple GPUs, AMD cards, vLLM and MLX runtimes, cloud fallback and authentication. Next in line are Radeon
163support, cost-based eviction instead of LRU (reload time à probability of reuse à priority), and quality tiers: under memory pressure, use a smaller variant of the same model. The Gemma 4 family fits that almost perfectly â 12B, E4B and E2B share one architecture and one prompt format.</p> 164<h2>The bottom line</h2> 165<p>The scheduling algorithm itself turned out to be the easy part. The hard part was accounting. The scheduler makes its decisions based on memory numbers, and nearly every serious bug came down to those numbers lying: an estimate based on file size, two inconsistent <code>nvidia-smi</code> calls, the memory of a process that hadn't finished unloading. A scheduler is only as good as its bookkeeping.</p> 166<p>GridCore is in beta and available to <a href="https://patreon.com/ZeroToMVP">Zero to MVP supporters on Patreon</a>. If you're working on a similar problem â on a single GPU or a whole rack â I'd like to compare approaches: <a href="/contact/">get in touch</a>.</p> 167`;export{e as frontmatter,t as html};
Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.