We made Stable Diffusion 2.1 run on an AMD RX 580 — 4.19× faster than it started, and we found out why it was painting blank pictures.

What you get

A 512×512 picture, 20 steps, on one eight-year-old graphics card that normally cannot run this software at all. Before: five minutes and twenty-five seconds. Now: one minute and eighteen seconds. Same card, same model, same code — different instructions inside it.

And it draws the same watts for a quarter of the time

A GPU does not get cheaper by being clever — it gets cheaper by finishing sooner. The card is pegged at full load for the whole render either way, so wall-clock time and electricity are the same quantity. Energy is power × time, and only the time changed.

Put in units you can act on: a hundred 512×512 renders went from 2.7 kWh to 0.65 kWh — about the difference between running a space heater for two hours and running it for half an hour, for identical output.

It did not just get faster — it started working

For a while every picture came out as a perfectly flat white rectangle. Not a bad picture. A blank one: one single colour, 8,746 bytes of file. That is the fingerprint of a number that has gone to infinity and been clipped on the way out.

Every number in this report comes from sd-rx580/evidence/bench.csv and is reproducible with sd-rx580/tools/bench.sh. Nothing here is estimated.

Why a graphics card is a rowing boat

A GPU does not run one instruction at a time. It runs them in fixed-size squads, and every member of the squad must move together — like a rowing crew in one boat. When the squad is told "everyone grab the fourth item", all 32 or all 64 rowers do it at once.

The catch: how many rowers are in the boat is a property of the hardware. Apple's own chips put 32 in a boat. This AMD RX 580 puts 64 in the boat. Software written by counting on 32 gets two possible endings:

• The wasteful one. It only ever calls 32 rowers. Half the boat sits idle, doing nothing, for the entire race.
• The dangerous one. It does the arithmetic for seat n assuming seats stop at 31. At seat 48 it asks for something that does not exist, and reads whatever happens to be next in memory.

Where the bug actually came from

The port had been told, in so many words, "pretend the boat has 32 seats". That works — the crew rows in half the boat and the numbers come out right. Unfortunately the other half of the code still believed it had 64 seats, and the two disagreed. The disagreement produced infinities, and infinity clipped to white.

So the first fix was not an optimisation. It was removing a lie: tell the software how many seats the boat really has. The white pictures disappeared, and the cat came back byte-identical to the older working build.

Does this matter beyond one old card?

Yes, and this is the part worth sending to anyone who thinks 64-lane hardware is ancient history.

Four bugs, in the order they were found

Each one was found by measurement, not by reading code and guessing. The order matters, because three of the four were invisible until the previous one was gone.

Bug 1 — the lie about the boat size

A width-32 clamp, added in two places. Symptom: pure white images, std exactly 0.00. The proof was a clean A/B — an older binary in the tree reported simdgroup width = 64 and rendered correctly; the newer one reported 32 and did not.

How we proved it was not the kernels: pointing the backend at the older kernel snapshot changed nothing; running the text encoder on CPU changed nothing; running the VAE decoder on CPU changed nothing. All three produced byte-identical white pixels, so the output was constant and the fault had to be in the C++.

Bug 2 — a number doing two jobs

Once images were correct, the conformance matrix showed the compressed weight formats — the ones squeezed into fewer bits to save memory — were badly wrong: 36 pass / 39 fail for q8_0, 37 / 43 for q4_0. The cause was one constant, NQ, being used for two unrelated things: how many weights one lane handles, and how many blocks the squad advances per round. At 32 seats those two numbers happen to be the same. At 64 they are not, and half the lanes read past the end of their block.

Splitting it into "lanes per block" and "blocks per round" fixed all four formats at once.

Bug 3 — the optimiser that never ran

This is the one that mattered most, and it hid for two hours.

We had written a tiled matrix multiplier — the thing that makes GPUs fast — and measured a solid 2× improvement. Then three more improvements each returned about 1%, which looked like a hardware ceiling. It was not. It was a ceiling on 30% of the work.

The fix to the measurement tool was trivial and embarrassing: a timing print that fired only on every 32nd event, when a single image produces far fewer than 32 events. Once it printed every time, the answer took ten seconds to read:

And the tiled multiplier had a guard: only for 32-bit float data. The convolutions — the entire decoder, and every convolution inside the generator — run in 16-bit data. They never used it. We had been optimising the part that was already fast.

Bug 4 — the obvious answer that lost

With a working tiled kernel, the obvious next step was a bigger tile: 32×32 instead of 16×16. It should have won, because a bigger tile reads the weights fewer times. It lost by 18%.

The reason is the same rowing boat. A 32×32 tile at one output per thread needs a crew of 1024. On this card that leaves room for barely two crews per engine, so there is almost nothing in flight to hide the memory delays. The fix was not a bigger crew — it was the same 32×32 tile held in registers, on a crew of 256. Same reuse, four times the concurrency. That is the last 13%.

The instrument that made this possible

None of the above was found by staring at code. It was found by a small test bench that runs the vendor's own conformance suite once per data format and prints a pass/fail matrix:

† Those two "before" figures come from the wider unfiltered run taken before the fixes — the plain-path filter used for every other number here did not exist yet, so they are indicative, not like-for-like. q4_0 and q8_0 are exact before/after on identical filters.

The bottom row is the honest part: four exotic formats are still wrong. They are listed, not hidden, and they do not affect Stable Diffusion because it never uses them. Every known failure is consolidated at the back →

We kept hitting the same bug in llama.cpp — so this is where we finally corner it

Everything above lives in a fork of stable-diffusion.cpp, which shares its compute layer with llama.cpp. That is not a coincidence we noticed afterwards. Running llama.cpp on this card kept producing the same wrong answers, across several attempts, and nothing we tried cleared it.

A diffusion pipeline was simply the easier place to catch the culprit: when the maths goes wrong there you get a visibly wrong picture, not a plausible-looking sentence. Chapter 1 is that capture. This chapter takes the same defect back to where it lives.

kernels/common.h:24
  #define N_SIMDWIDTH 32   // default; ggml_metal_device overrides it with the real width

ggml-metal-device.m:470
  if (simd_width > 0 && simd_width != 32) {
      [prep setObject:... forKey:@"N_SIMDWIDTH"];   // adopts the real width
  }

kernels/mul_mv.metal:259
  const short ix = (tiisg/(NW/NQ));
  const short il = (tiisg%(NW/NQ))*8;              // NW = N_SIMDWIDTH

Read those three lines together in plain terms. The card tells the software how many rowers are in the boat, and the software listens — good. But then it decides which part of a box to fetch by dividing the seat number, and with 64 seats that division hands out positions the box does not contain.

Weights live in small boxes of 32 numbers. Two rowers finish a box: sixteen numbers each. The code assumed four rowers would share it — which is only true when the boat has 32 seats. With 64 seats, two of every four rowers reach past the end of their box and pick up whatever is lying next to it in memory. Nothing crashes. The arithmetic still runs. The answer is just quietly wrong.

static bool ggml_metal_supports_mul_mat_wide(...) {
    // q4_0/q4_1/q5_0/q5_1 still route through mul_mv_ext, which is not yet correct at 64
    // lanes (0/121, 0/99, 0/63, 0/56 on MUL_MAT) even with nypsg and the dispatch width
    // fixed - keep them on the CPU path until that is resolved.
    case GGML_TYPE_Q4_K: ...

How long has this been open?

Longer than a bug this simple should survive — and mostly because each report landed as an isolated mystery instead of one named defect.

So: reported in 2023, closed as stale, rediscovered in 2026. Three years is not because the fix is hard — it is three small edits — but because nobody owned the problem long enough to reproduce it twice.

Credit where it is due: our fork already carries the first two fixes from #19601 — concurrency disabled when the GPU has no unified memory, and the simd_max half→float cast. That work is why this card renders at all. The third item is different in kind: q4_0/q4_1/q5_0/q5_1 are not broken by accident. They are excluded on purpose, in a comment that asks for exactly this fix.

The patch, in three ideas

// 1. Derive "lanes per block" from the block, never from the wave width.
constexpr short NLB = QK4_0/NQ;   // 32 quants / 16 per lane = 2 lanes per block
constexpr short NBL = NW/NLB;     // blocks advanced per round: 16 at 32 lanes, 32 at 64
const short ix = tiisg/NLB;
const short il = (tiisg%NLB)*8;

// 2. Advance by the number of blocks actually covered per round.
for (int ib = ib0; ib < nb; ib += NBL) { ... yb += QK4_0*NBL; }

// 3. Same treatment for q8_0, which has the identical "NQ does two jobs" defect.

At 32 lanes these reduce to exactly the current expressions, so Apple Silicon cannot regress — that is the property that makes the patch reviewable.

The performance half

The same card was also slow, for an unrelated reason. When the graphics chip has no built-in shortcut for multiplying tables of numbers — and this one does not — the software falls back to a method that re-reads the entire weight table once for every single output column. That is like re-reading the whole recipe before cutting each individual slice of cake.

The fix is the obvious one in kitchen terms: read the recipe once, then cut a whole tray of slices from that one reading. In this card's case that alone was worth more than doubling the speed. The engineering detail is in llama-cpp/ALGORITHM.md for anyone who wants it.

How to check it yourself

# On any device where threadExecutionWidth != 32:
./build/bin/test-backend-ops -b MTL0 -o MUL_MAT -p type_a=q4_0
./build/bin/test-backend-ops -b MTL0 -o MUL_MAT -p type_a=q8_0

# Note: -p splits on commas into key=value pairs, so shape filters such as
# bs=[1,1] cannot be expressed there - filter the printed case strings instead.
# Export GGML_METAL_PATH_RESOURCES to the source tree, or the backend compiles a
# stale copy of the kernels from build/bin/kernels and your edit does nothing.

Patch, facts and adoption notes: wave64-report/llama-cpp/. Deliberately no PR draft — llama.cpp prohibits AI-written PR descriptions, so the submission has to be written by the person who did the work.

The details, without the kitchen words

Where the code lives

Reproduce it

cd sd-rx580/Vendor/stable-diffusion.cpp
cmake --build build --target sd-cli test-backend-ops -j8   # skip the .air targets:
                                                          # xcrun metal is not installed

cd ../../tools
./wave_probe                       # dispatch-width semantics on this GPU
./matrix.py --plain f32 f16        # the gate: expect 0 fail
PROMPT="a photo of a cat" STEPS=20 SEED=42 ./bench.sh mytest   # s/it + image stats

Numbers as measured

Things that will waste your afternoon

• The kernel copy. Without GGML_METAL_PATH_RESOURCES, the backend compiles build/bin/kernels/*.metal — a build-time snapshot. Edits under ggml/src/ silently never reach the GPU.
• -p is not a regex. It splits on commas into key=value constraints, so bs=[1,1] becomes two broken tokens. Filter in Python instead.
• Shader compilation is not free. ~50–80 s per changed kernel source, per process. Batch your kernel edits.
• xcrun metal is missing without full Xcode, so the metallib targets fail. Build the binaries only.
• A gate that can never pass is not a gate. An unfiltered matrix is dominated by broadcast/permute shapes no diffusion model emits.

Known failures — kept deliberately

Everything below is still broken, still unmeasured, or still an assumption. It is gathered in one place, at the back, so it cannot be missed and no reader has to discover it alone.

Two of these deserve a sentence rather than a table cell.

Flash attention is the failure that costs you something. The port wraps every fa_vec_* instantiation in #if N_SIMDWIDTH == 32, so at width 64 the entire vector flash-attention family compiles out and the UNet falls back to explicit softmax. Porting those kernels is a prerequisite for flash attention to exist here at all — not a tuning knob to turn later.

The quantised formats nobody uses are still wrong. It would have been easy to delete four rows from a table and ship a report that looks complete. They stay because a matrix with holes in it, honestly labelled, is worth more than a green one that quietly omits its failures.

If something in this report is wrong, the most likely place is one of the eleven rows above.