DeepSeek-V4-Flash-0731 at ~40–50 tok/s on 2× Radeon AI PRO R9700 with the affinity engine (prebuilt quant + fixes)

[u/neuromacmd]

I've been running DeepSeek-V4-Flash-0731 (custom quant 2.875bpw) on two Radeon AI PRO R9700s (32 GB each, 192 GB system RAM) using affinity(https://codeberg.org/StillDeadcode/affinity), an inference engine written by **Yoshi Exeler (StillDeadcode)** specifically for DeepSeek-V4-Flash on one or two RDNA4 cards. All the credit for the engine goes to them. It keeps the hot experts on the GPUs, streams the rest from host RAM, and runs a DeepSeek-native speculative draft. If you have R9700s, go look at it.

What I added:

- A prebuilt 0731 quant, so you can skip the 149 GB download and the ~2.5 h quantize: [neuromaniacmd/DeepSeek-V4-Flash-0731-affinity-2.875bpw](https://huggingface.co/neuromaniacmd/DeepSeek-V4-Flash-0731-affinity-2.875bpw). It was made with affinity's own quantizer: routed experts at 2.875 bpw, the teamblobfish imatrix, dense weights in FP8, plus 0731's bundled draft. It loads only in affinity, not llama.cpp or vLLM.
- A fork with a few stability fixes ([github.com/neuromaniacMD/affinity](https://github.com/neuromaniacMD/affinity)). The main one: on 2×32 GB cards the server would start producing fluent garbage after the first long prompt and never recover, caused by hugepage collapse under a GPU mapping with too little free VRAM. The fixes are also submitted upstream as PRs. There's a Docker build on TheRock ROCm 7.14, so the host needs no ROCm install.

Numbers on my box (2× R9700, 262K context fully allocated at load):

- Prefill: ~1,260–1,360 tok/s at 4K–64K context
- Decode: ~40–50 tok/s on long structured answers, ~31–35 on free-form prose and reasoning. It depends on how often the draft's guesses are accepted.
- Needle-in-a-haystack 3/3 at 4K/16K/64K. A multi-turn agentic task (Excel → DuckDB via shell tools, 15 turns) passed every check, with no malformed tool calls.

Caveats:

- RDNA4 (gfx1201) only, and two cards max.
- Use the model card's sampling (temp 1.0). At temp 0.6 this quant fell into reasoning loops for me.
- You need ~64 GB+ of RAM for the experts that don't fit on the cards.
- There's also an experimental DeepSeek-V4.1-Flash branch. It's correct now, but only ~14 tok/s on two cards.

Happy to answer questions. Engine questions really belong with the upstream project.

UPDATE / EDIT (BetterBench, 5-pass quick run):

Edit: BetterBench numbers (GGZ14/BetterBench v0.6.0, quick run = 5 passes per category, so treat it as indicative; same box, 2× R9700, via llama-swap)

Decode, single stream, median tok/s:

math	json	summarization	file_edit	code	reasoning	chat	prose
47.1	43.4	40.5	40.4	34.8	33.5	33.1	26.8

• Weighted combined: 35.8 tok/s, time to first token ~0.7 s in every category, update p99 ~110 ms.
• Prefill (cold cache): 1,150 tok/s at 1.6K → 1,900 at 6K → 2,090 at 12K → ~2,200 at 24K–47K.

Caveats: - Thinking was on. 60 % of runs hit the corpus's max_tokens cap while still reasoning, so these are mostly thinking-phase speeds, and time-to-answer isn't measurable on this corpus. - Sampling was BetterBench's default (temp 0.7). Speed follows draft acceptance: structured output is fastest, open prose slowest. - Concurrency sweep skipped: affinity serves one sequence at a time, so it would only measure queueing. - My own long-document prefill measured lower (~1.3K tok/s). BetterBench's filler text is probably friendlier to the expert cache than real documents.