Output you can
trust again.

Only recipes ported for this kitchen are allowed

ggml keeps a whitelist for the wide-kitchen case. Anything not on it is refused and cooked the slow way, on the CPU:

// on devices without simdgroup-reduction the kernels run with the real
  // simdgroup width; only the types ported for that width are allowed on MUL_MAT
  static bool ggml_metal_supports_mul_mat_wide(const ggml_tensor * op) {
      switch (op->src[0]->type) {
          case GGML_TYPE_Q4_0: ... case GGML_TYPE_Q4_K: return true;
          default: return false;
      }
  }

On this card simdgroup reduction = false, so the wide path is the only path — and the list was short.

Not a crash. A wrong answer.

A wrong dish never announces itself; it just tastes wrong. That is the worst kind of bug in a kitchen, because nothing stops and everything downstream is quietly wrong.

Count the cooks in the row

Every one of those kernels had the same hidden constant: four cooks to a tray. That is what "32 lanes ÷ 8" equals, and it was written as a literal. With 64 lanes the stride and the pointer advance must both be eight — so the rule became "count the row, then divide":

const int n_lanes_per_tray = N_SIMDWIDTH / 8;   // 4 on Apple, 8 here

Two lines per kernel in some cases, three in others. The reason it stayed hidden: at 32 lanes the new expression equals the old literal, so Apple's kitchen — where everyone tests — cannot tell the difference. Only a 64-lane row exposes it.

Eight formats, one still banned

q3_K is the one left. It does not use the same tray split — its index carries a different role — and the q4_K pattern applied to it made things worse, so the gate went back. Writing that down is cheaper than rediscovering it.

The card was never the limit

Correct kernels turn an unanswerable question into a measurable one: how fast is this card, really? A decode-shaped matvec (m=4096, n=1, k=14336) has to move 117.4 MB. It does that in 524.9 µs — 224 GB/s, 87% of the card's 256 GB/s. The silicon could always do this. The recipes could not.

Now the honest reading. f16 reaches 224 GB/s, but it is also reading 3.5x more bytes in the same call, and a large transfer is more efficient than a small one. q4_K at ~100 GB/s is not near a defect: a follow-up probe measured 104.7 GB/s as the streaming ceiling for a 33 MB read on this card, and 113.9 GB/s was the best any simple kernel reached. So q4_K sits roughly where a read that size tops out - and q5_K, at 48 GB/s, is the one genuinely twice off the pace.

Two follow-ups belong here because they killed good-sounding ideas. Splitting the q4_K scales into their own stream - the obvious "repack the file" fix - measured 42 GB/s against 79: the native interleaved block is the better layout. And giving each thread four times the bytes helped a naive probe (78.6 -> 113.9 GB/s) while the shipped kernel already reaches 104.7, so real headroom is about 9%, not 45%.

So unblocking did not make the card fast. It made the card measurable — and a measurement is what tells you which recipe to fix next. Everything in the next chapter starts from this table.

One thing here is speed, and it has to be attributed carefully: four of these formats used to fall back to the CPU because their kernels never reached the card. Sending that work to the GPU changes where it runs — but it was never timed on its own. The four seconds measured in the next chapter belong to that rerouting and to a tile change shipped in the same build, so neither may claim the number alone.

The only limit left is video memory

Here is the part that matters more than any speed number. Q4_K_M is what most GGUF models ship as. Before the fix it was the worst case on this card: one quantised format answered wrong, four never reached the card at all and quietly ran on the CPU. So the fix did not rescue a recipe. It opened the shelf — the standard model files in circulation became usable, installable and improvable on a used eight-year-old card.

With formats no longer the question, the limit becomes arithmetic: weight bytes plus context. Measured on this card, with the model loaded and the server running:

The largest thing we actually served is a 7.9B model at Q6_K, in 6830 MiB — 6.7 of the card's 8 GiB. At f16 the same model would want about 16 GiB, which is a different card. That is the whole point in one line: the unlock moved the wall from "this card refuses the format" to "this card has 8 GiB", and a memory budget is a wall a person can plan around.

Two honest riders. Fitting is not the same as being good — the 4B fit comfortably and produced bad output, so size is not quality and this page does not measure quality. And context is cheaper than people fear at this scale, but not free: the whole server with the 1.7B at 16k measured 1700 MiB, roughly 90 MiB above the model alone. It grows with length and with model size, so a long chat and a large model still compete for the same eight gigabytes.

What this chapter has not done

The page above is what is proven. This is what is not, so nobody mistakes the two.

• Nothing here is faster. Correctness was the claim; the CPU-to-GPU reroute was never timed on its own, and the four seconds in the next chapter are shared with a tile change.
• q5_K still runs at 48 GB/s, twice off the streaming ceiling measured on this card. Known, not explained, not fixed.
• q2_K and q3_K keep the same defect, still gated, still hidden by the CPU fallback.
• The 32-lane argument is algebraic, not measured — no Apple GPU is available here.
• One shape, one card. The ceiling comes from a single 33 MB read on one RX 580.

Honest boundaries

Correctness only. Four formats used to fall back to the CPU and now run on the card, but that rerouting was never timed on its own — so this chapter claims no speed. What it claims is narrower and more useful: a dish the kitchen can cook at all. Speed is the next chapter, and it is a different bug.

What this cost, and what it is worth

1,292 correctness checks across eight quantised formats, eleven bandwidth measurements, and one afternoon discovering that two constants were doing one job. The fix is thirty lines.

What it is worth: four formats that used to quietly run on the CPU now run on the card, and the card finally has a measured ceiling instead of a mystery.

Keep the AI cooking — any amount from $3

Why the middle tier: this is the chapter that makes the shelf loadable at all, but it claims no speed of its own — $6 is for a one-hour note, and $53 is for a chapter that hands you back a working day.

A suggestion, not a price. No ads, no tracking, no newsletter, and no third-party script: this page renders the same with the network switched off. And if money is tight, a fair trade is worth just as much: send this page to one person with an old GPU gathering dust.