Chapter 2.1 taught the kitchen to cook these dishes at all. It was still slow, because to answer 512 questions it walked back to the shelf 512 times. Two changes fixed that: every cook now carries eight trays instead of two, and every row of trays has one empty slot at the end so nobody queues at the same tray.
The fast way to serve a table of 512 is to open the recipe book once and answer all of them. The kitchen was opening it once per question — measured, not guessed: 512 × 33 MB of weights at about 108 GB/s.
A tile fixes that. One team lifts a 64-row × 64-column block of the answer, unpacks the weights for it into the shared counter once, and serves every column from that. The unpacking is the expensive part, and it now happens once per tile instead of once per row.
The counter hands out trays in 32-bank rounds, cycling every 128 bytes. A row of 64 plates is 256 bytes — exactly two rounds — so every cook reaching for "plate k of their row" grabbed the same tray. Thirty-two cooks, one tray, a queue, while thirty-one trays watched.
threadgroup float sA[BM][BK]; // 64 floats = 256 bytes = two bank rounds threadgroup float sA[BM][BK + 1]; // 65 floats: rows no longer line up
Cost: 31 floats of shared counter per tile. Gain at model level: 133.7 → 197.7 t/s on the same build, measured by flipping one environment variable.
Not every win is a kernel. This card reports has bfloat = false, so any dish written on a bf16 card is handed to the slow kitchen — once per layer. The model file had 29 of them: the token embedding and 28 attention gates, each 2048 × 8 — sixteen kilobytes each, costing half the decode rate.
Re-encoding them as f16 keeps every other tensor type identical (q8_0 29, q4_K 84, q6_K 28, f32 57) and the file the same size, 1.65 GiB. Nothing is lost: f16 carries ten mantissa bits, bf16 seven.
| agent task: open a repo, read a README, answer in two bullets | before | after |
|---|---|---|
| cold run | 128 s | 66 s |
| prompt evaluation | 94.9 t/s | 154.7 t/s |
| generation | 16.6 t/s | 29.4 t/s |
Prompt processing was 86% of that task, which is why the tile shows up so plainly in the wall clock.
q8_0 got the same treatment — 204 ms → 44 ms in isolation — and then turned an English prompt into Chinese and ran 4096 tokens without stopping. It sat behind a flag for a day, written off as an integration mystery.
It was not a mystery, and it was not the kernel. The dispatch chose the tile geometry from a type list that named q4_K and q6_K but forgot q8_0, so the q8_0 kernel was launched with the q6_K shape: 16 rows per block and 128 threads, while the kernel stages 64 rows with 256 threads. Half the tile was never written, and uninitialised shared memory reads back as -inf — which is how a prompt becomes Chinese.
One line adds q8_0 to that set. Then: 117/117 correctness cases, model output identical to the tile-off run, and prefill 296.05 → 372.44 t/s (+26%) with decode unchanged at 40.7. q4_K (123/123) and q6_K (62/62) were unaffected throughout.
The lesson is sharper than "test more". A private harness had already certified this kernel bit-exact — using its own dispatch. The model calls a different code path, and that path fed the kernel a shape it was never written for. Test the dispatch a model actually uses, or the test proves something nobody runs.
The diagnosis came later and it corrects what this page first said. In the full harness the tile fails at m=2048, n=256, k=2048 — the model's own attention-projection shape — returning -inf where the CPU says 20.0. Neighbouring widths pass (n=32, 64, 128 and 512 are fine), small heights return NaN, and it fails three runs out of three. Disabling fusion or concurrency changes nothing, so it is not a race.
Earlier notes claimed the kernel matched the reference at that shape. It does not; that result came from a private harness with different buffer setup. That is the real lesson, and it has now cost this project three separate days: a green private harness is not a green real one. The tile stays behind a flag until one shape, the one a model actually calls, comes back finite and correct.
Two bottlenecks, and they never move together. Prefill is arithmetic: 6,800 tokens in goes through big matrix multiplies, so prefill tracks FLOPS. Decode is delivery: one token at a time re-reads every weight, so decode tracks bytes per token against 256 GB/s. A model that is fast at one can be slow at the other, and "which model is fastest" has no answer until you say which half of the work dominates.
| model | shape | prefill | decode | J/token | VRAM |
|---|---|---|---|---|---|
| 1.7B tiled | dense 1.7B | 295.7 | 39.7 | 5.9 | 1.6 GiB |
| Ling 7.9B | MoE, 1.3B active | 148.4 | 30.1 | 8.6 | 6.2 GiB |
| 8B-A1B | MoE, 1B active | 85.2 | 41.8 | 6.3 | 5.4 GiB |
| 1.7B stock | dense 1.7B | 91.7 | 21.1 | 12.0 | 1.6 GiB |
| 4B | dense 4B | 32.9 | 12.9 | 22.1 | 3.3 GiB |
| 9B | dense 9B | 12.8 | 6.0 | 43.5 | 5.6 GiB |
Every row in that table is loadable because the quantised kernels are correct. That is the quiet benefit of the previous chapter: the hard limit is no longer the format, it is 8 GiB of VRAM — and a 7.9B model at Q6_K was measured running inside 6.7 of those 8 GiB.
Why the sparse model wins decode and loses the job: it only touches about one billion parameters per token, so delivery is cheap. But it must still read the whole prompt before it can answer, and reading is arithmetic — 85 t/s against 296. On a read-then-answer task that is the difference between 66 seconds and 605.
The dense 4B is the trap on this card: twice the parameters of the 1.7B and worse in every column, including 22.1 J/token. The 9B fits, runs, and costs seven times the energy per token for a third of the speed.
Bench rates are one thing; the task is another, and two of these models now have both. Run through the same agent task, same harness, same hour: Spark 1.7B finished in 93.1 s, Ling 7.9B in 219.4 s, and the 8B MoE in 278.6 s — even though the MoE is the fastest decoder of the three at 39.7 t/s.
That last row is the lesson in one number. The MoE wrote 3,073 tokens in its second turn where Spark wrote 263, so almost 90 seconds of its 279 were its own verbosity. Decode speed is a rate; task time is a rate times how much the model decides to say. A fast talker can still be the slowest worker, and a table of token rates cannot show it.
And one more number, because it is the honest one: after the q8_0 dispatch fix below, the same task ran 87.0, 89.0 and 97.1 s across three attempts. Prefill improved in both turns (203 -> 236 t/s, then 138 -> 152), but the wall clock is dominated by how much the model decides to say: the 97 s run wrote 667 tokens where the 87 s run wrote 365. The kernel got faster; the model's mood is not a controlled variable.
The prompts were 7,233-7,530 tokens, so these three are the closest thing to an apples-to-apples comparison on this page. (It is also why 93 s today is not a regression from the 66 s ladder: that ladder used a 6,804-token prompt.)
Format is a second dial, and it does not buy speed here. The same 1.7B as Q8_0 decodes at 39.97 t/s against Q4_K_M's 39.7 — identical to the noise — while the file grows from 1.64 to 1.96 GiB and peak VRAM from 1609 to 2170 MiB. Prefill drops to 89 t/s because the q8_0 tile is still gated. So Q8 is a precision choice, not a speed one, and on 8 GiB it costs you the bigger model: Q4 ran a 7.9B inside 6830 MiB, and Q8 at that size would need about 11.
The benchmark is a real agent task, not a chat prompt: "list the files here, read README.md, summarise it in two concise bullet points", run through a local agent against the local server. It is two turns by design, because that is the shape of real use.
| turn | what the server sees | tokens |
|---|---|---|
| 1 | harness prompt + tool schema + question | 6,804 |
| tool call | the model asks to read the file | 25 |
| 2 | previous turn + file contents | 3,161 |
| answer | two bullets | ~200–250 |
That shape makes prefill the bill: 6,804 tokens at about 100 t/s is 67 seconds before the model says anything. It also includes a real tool round trip and it is identical across builds, so a difference is the build and nothing else.
| when | what changed | task |
|---|---|---|
| 11-02 | four models evaluated | 216 / 363 / ~340 / 605 s |
| 11-04 16:24 | lane split + 4 missing cases | 132 → 128 s |
| 11-04 17:53 | tiled q4_K | 112 s |
| 11-04 18:07 | the +1 padding | 86 s |
| 11-04 18:12 | model re-encoded f16 | 79 s |
| 11-04 18:40 | tiled q6_K | 66 s |
Limits, stated plainly: these are single readings, not averages of three. Decode on the task swung 30.4 → 23.8 t/s on the same build between runs while prefill held steady. And the model table above comes from a shorter prompt than the 128 → 66 s ladder — compare models in the table with each other, never with 66 s.
The page above is what is proven. This is what is not, so nobody mistakes the two.
These are this card's numbers on this build. The tile helps where the matrix multiply is the workload — prompt processing, long context, big batches. It does nothing for a decode step that is bound by dispatch, and nothing at all for a pipeline whose time is spent elsewhere, as the image side found to its cost.
50 benchmark runs, 60 power traces, six kernels written, four shipped and one thrown away — including a 4.6× optimisation that produced nonsense at full size.
What it is worth: a 128-second job now takes 66, and a 1.7B model that decoded 20 tokens a second decodes 39 on the same card.
Keep the AI cooking — any amount from $3
Why the top tier: six kernels written, four shipped, one thrown away, and a 48% cut in a task you run all day. A day of your time is worth more than this — if it saved you one, this is the button.
A suggestion, not a price. No ads, no tracking, no newsletter, and no third-party script: this page renders the same with the network switched off. And if money is tight, a fair trade is worth just as much: send this page to one person with an old GPU gathering dust.