DeepSeek-V4-Flash-0731 at ~40–50 tok/s on 2× Radeon AI PRO R9700 with the affinity engine (prebuilt quant + fixes)

[u/neuromacmd]

I've been running DeepSeek-V4-Flash-0731 (custom quant 2.875bpw) on two Radeon AI PRO R9700s (32 GB each, 192 GB system RAM) using affinity(https://codeberg.org/StillDeadcode/affinity), an inference engine written by **Yoshi Exeler (StillDeadcode)** specifically for DeepSeek-V4-Flash on one or two RDNA4 cards. All the credit for the engine goes to them. It keeps the hot experts on the GPUs, streams the rest from host RAM, and runs a DeepSeek-native speculative draft. If you have R9700s, go look at it.

What I added:

- A prebuilt 0731 quant, so you can skip the 149 GB download and the ~2.5 h quantize: [neuromaniacmd/DeepSeek-V4-Flash-0731-affinity-2.875bpw](https://huggingface.co/neuromaniacmd/DeepSeek-V4-Flash-0731-affinity-2.875bpw). It was made with affinity's own quantizer: routed experts at 2.875 bpw, the teamblobfish imatrix, dense weights in FP8, plus 0731's bundled draft. It loads only in affinity, not llama.cpp or vLLM.
- A fork with a few stability fixes ([github.com/neuromaniacMD/affinity](https://github.com/neuromaniacMD/affinity)). The main one: on 2×32 GB cards the server would start producing fluent garbage after the first long prompt and never recover, caused by hugepage collapse under a GPU mapping with too little free VRAM. The fixes are also submitted upstream as PRs. There's a Docker build on TheRock ROCm 7.14, so the host needs no ROCm install.

Numbers on my box (2× R9700, 262K context fully allocated at load):

- Prefill: ~1,260–1,360 tok/s at 4K–64K context
- Decode: ~40–50 tok/s on long structured answers, ~31–35 on free-form prose and reasoning. It depends on how often the draft's guesses are accepted.
- Needle-in-a-haystack 3/3 at 4K/16K/64K. A multi-turn agentic task (Excel → DuckDB via shell tools, 15 turns) passed every check, with no malformed tool calls.

Caveats:

- RDNA4 (gfx1201) only, and two cards max.
- Use the model card's sampling (temp 1.0). At temp 0.6 this quant fell into reasoning loops for me.
- You need ~64 GB+ of RAM for the experts that don't fit on the cards.
- There's also an experimental DeepSeek-V4.1-Flash branch. It's correct now, but only ~14 tok/s on two cards.

Happy to answer questions. Engine questions really belong with the upstream project.

UPDATE / EDIT (BetterBench, 5-pass quick run):

Edit: BetterBench numbers (GGZ14/BetterBench v0.6.0, quick run = 5 passes per category, so treat it as indicative; same box, 2× R9700, via llama-swap)

Decode, single stream, median tok/s:

math	json	summarization	file_edit	code	reasoning	chat	prose
47.1	43.4	40.5	40.4	34.8	33.5	33.1	26.8

• Weighted combined: 35.8 tok/s, time to first token ~0.7 s in every category, update p99 ~110 ms.
• Prefill (cold cache): 1,150 tok/s at 1.6K → 1,900 at 6K → 2,090 at 12K → ~2,200 at 24K–47K.

Caveats: - Thinking was on. 60 % of runs hit the corpus's max_tokens cap while still reasoning, so these are mostly thinking-phase speeds, and time-to-answer isn't measurable on this corpus. - Sampling was BetterBench's default (temp 0.7). Speed follows draft acceptance: structured output is fastest, open prose slowest. - Concurrency sweep skipped: affinity serves one sequence at a time, so it would only measure queueing. - My own long-document prefill measured lower (~1.3K tok/s). BetterBench's filler text is probably friendlier to the expert cache than real documents.

[u/myholeisstinky]

Gotta make the level of quant much more prominent, else the rest of it is meaningless

[u/SkibidiOhio67Tax]

Are you using this agentically? What do your concurrency lanes look like?

[u/neuromacmd → u/SkibidiOhio67Tax]

Yes, some. I drive it from pi, a coding-agent CLI, through llama-swap. On a short agentic task (convert a multi-sheet Excel workbook into DuckDB using shell tools) it took 15 turns and 17 tool calls, with zero malformed tool calls, and every check against the known answers passed. That was about 5 minutes, ~20 s per turn. The earlier V4-Flash build of this lane also passed a much longer run: 65 turns and 71 tool calls, also clean, but that took ~90 minutes. It's correct on long tool loops, just not fast.

Concurrency is the weak spot: affinity serves one sequence at a time, and a second request just queues behind the first. So it's my "one careful agent / deep reasoning" lane, not a multi-user server. For concurrency I swap other models onto the same two R9700s through llama-swap, e.g. Qwen3.8-27B on vLLM with tensor parallelism at 4 sequences. A third card (a W7900) runs a small always-on model for quick parallel stuff, so everyday requests don't wait on DeepSeek.

[u/Kasatka06]

Do you know why the stilldeadcode vllm limits to maximum of 2xR9700 ? I saw the repo and i dont think they mention recipe for 4xR9700.

[u/neuromacmd → u/Kasatka06]

Small correction first: affinity isn't vLLM. It's a standalone C++/HIP engine written just for DeepSeek-V4-Flash. StillDeadcode's vLLM work is a separate project, vllm-radiance.

The two-card cap is on purpose. The engine refuses to start with more than two GPUs. The author tested the pieces individually at four ranks (expert split, all-reduce, attention) and they all checked out, but the engine as a whole still produced repeated-token garbage past two ranks and they couldn't find why. So they block it rather than ship wrong output. They also didn't have a box with more than two cards, so the failure was seen by simulating 4 ranks on 2 cards. AFF_ALLOW_NRANK=1 overrides the check, but the README says not to trust the output.

Since then: I've been porting affinity to CUDA on a 4-GPU box (for DeepSeek-V4.1). On 4 real cards I found and fixed two bugs that broke it at 4 ranks. It now matches single-card output and is faster than 2 cards. But those bugs only trigger with V4.1's expert size. V4-Flash splits cleanly 4 ways, so they're probably not what the author hit. I also suspect the 4-on-2 simulation itself: on my box it gave different results from run to run. Bottom line: nobody has actually tried V4-Flash on 4 real R9700s yet.

If you have 4× R9700 and want to try it: all four need to be on CPU lanes with P2P working between every pair, because the all-reduce writes directly into each peer card. Chipset slots don't do P2P. Three cards won't split evenly (64 attention heads). If you run it with AFF_ALLOW_NRANK=1, compare against --gpus 2 on the same prompt with the same seed. I'd love to see the result.

[u/Kasatka06 → u/neuromacmd]

Thanks for the correction for affinity engine. I have 4x3090 and already enabled p2p using some gpu driver patch. I'am always curious how to enable p2p on R9700 would you share how p2p works like its using driver patch or what method to enable it, i tried to do some research but still not have definitive answer.

I dont have R9700 now but i'am thinking to replace my 4x3090 for 4xR9700, 3090 is a beast but its aging

[u/neuromacmd → u/Kasatka06]

For the R9700 the only way I know is to use a supported platform which in my case has been an AMD chipset - I have made it work on b850, x670e, x570e and sp3 epyc platform. I have not had to do anything special to make it work. I tried running them on z890 but there was a significant performance hit. I think non consumer intel platforms support p2p on this cards. There are hacks to make it work on non supported platforms but I am not sure how well they work.

[u/Kasatka06 → u/neuromacmd]

very informative ! i also have an SP3 platform (Huananzhi D12) so thats should be automaticaly work.

[u/brainchillzZ → u/neuromacmd]

I'm sorry I didn't see this before I responded ... I have a real four card box I'll just give it a try and see what I can see.

[u/brainchillzZ → u/brainchillzZ]

https://www.reddit.com/r/LocalLLaMA/comments/1x70e5r/deepseekv4flash0731_on_4x_radeon_ai_pro_r9700/

[u/brainchillzZ → u/Kasatka06]

the two card cap I assume is because just like the radiance project they use a custom upstream all-reduce that is only setup to use 2 or maybe 3 cards and disables all-reduce if more cards are used. I submitted patches to enable the use of 4 cards (in my case with radiance) and I'm using radiance with 4 cards locally myself.

https://github.com/brainchillz/r9700-tp4-allreduce
https://github.com/jdrews/vllm-mxfp4/pull/2

Edit: Sorry I didn't see your previous comment already that contradicts some of this. I'll give it a live try and see what I can see on a real four card box. It's downloading now.

[u/brainchillzZ → u/brainchillzZ]

Tonight or tomorrow I will give this a shot and see if I can adapt what I did for radiance to work with this on my 4 card box as well ... without even looking at the code just from reading the description I'm betting it will be very similar.

[u/brainchillzZ → u/Kasatka06]

So far my four node run is producing coherent results out of the box with pretty much no alterations to what u/neomacmd posted above saving ignoring the ranks error so it goes past the error and loads ... it looks like the issue was just with the 4 ranks simulation and it actually works on four real cards. https://www.reddit.com/r/LocalLLaMA/comments/1x70e5r/deepseekv4flash0731_on_4x_radeon_ai_pro_r9700/

[u/DiscipleofDeceit666]

So QFN or Deepseek?

[u/neuromacmd → u/DiscipleofDeceit666]

Which one is better? I think Deepseek is overall a better model, but Qwen Flash Next runs about double the speed of DS v4 flash. Different use scenarios, I guess.

[u/Mirayum → u/neuromacmd]

In my coding tests (same quants) QFN was clearly ahead though

[u/neuromacmd → u/Mirayum]

What harness? I don’t do development. My use case is data analytics and database analysis and regulatory research/compliance in documents

[u/Mirayum → u/neuromacmd]

I tested on DeepSeek Harness and Opencode