r/LocalLLaMA • u/Express_Quail_1493 • 9h ago

Question | Help Does strata on multi-gpu without nvlink worth it?

GPU- i have asymetric architectures - Titan RTX(a 2018 card released same time as 2080ti) and i have a RTX 3090. Both of these cards are 24gb the TitanRTX does not have flashAttention-2 only standard flash attention

SystemRam - i have about 48gb of Free SystemRam to spare. the rest is used up by my chrome VsCode and docker instances. 6000mhz ramspeed.

Im getting 32tok/s on qwen3.8-27b running on a single 3090 at 300k context the Titan Rtx is pretty much empty and just sitting there.

but im wondering is strata designed for multigpu will it massively decrease my prefill speed on multigpu? Anyone tested?

is it faster on a single 3090 and the rest in system ram or would it be faster on dual cards even with my specific caveats?

0 Upvotes

50% Upvoted

10 comments sorted by

5

u/voogamer 9h ago

Go test it. Only you can decide if it's worth it.

I have 4x4090. I see a big increase in prefill/tg if I use Strata with Q4XL on 1 GPU or 4. Definitely not 4 times as fast, but the increase is there.

I really should do a speed benchmark at some point, to compare speeds on 1, 2, 3, or 4 GPUs, and compare it with power usage too.

2

u/suprjami 8h ago

Strata prompt processing is a lot faster than llama.cpp running the same model. For me:

  • llama.cpp = ~200 pp/sec
  • Strata on one GPU = ~1200 pp/sec
  • Strata on two GPUs = ~1800 pp/sec

I have 3080 20G so no NVLink, one card is in a PCIe x4 slot.

1

u/Steus_au 7h ago

same here two 3080s, I have switched from iq4 to q4 today (0.1.44.1 supports it now) and prefill is still 1200 (it was 2200 with iq4 though)

1

u/suprjami 6h ago

I need to do a really long prompt (90k) to hit 1800. However on a prompt like 20k-50k I hit like 1600.

I think this depends somewhat on CPU speed. I have a i7-14700 with "--pool-workers", "7" in model config. The CPU has 8 P-cores.

1

u/egnegn1 5h ago

I run Strata with 2xRTX5080 and IQ3_S quant and get PP 5000 t/s and TG 150+ t/s.

Strata does some interesting streaming of kv cache and experts which hides parts of impact of slower memory.

Just try it. Of course, Strata works better with more modern cards and faster CPU and PCIe speed. I don't know whether P2P is supported with this old cards.

1

u/Perfect-Campaign9551 3h ago

I thought Strata was actually tuned especially for 3090s, I wouldn't expect it to do anything with ancient tech like a RTX Titan

1

u/Constant-Simple-1234 2h ago

Yes. Go for it. 8x pcie 3.0 It does not saturate it.

1

u/Moist-Tumbleweed2875 llama.cpp 27m ago

yes , without nvlink still work well ,following run on PCIE 4.0x8 +4.0x4 +3.0x4 :

1

u/Excellent-Issue-5956 13m ago

No NVLink here either, two 5070 Tis with the second one on an OCuLink dock at Gen4 x4. On stock Strata with the ISTA IQ3_XXS of Flash-Next, decode went from about 59 tok/s on one card to 97 on two. Prefill went the other way, about 2K tok/s on one and 1.6K on two.

suprjami has an x4 card too and got faster prefill on two, so I'm not sure what's different on mine. Mine runs under WSL2 on Windows, which might matter. That's Next and not the 27B though, so I can't say how the dense model splits.

1

u/JinsooJinsoo 0m ago

Look into hyperqwen for the 3090, I get like 90 tok/s at 131k context, I haven’t tested higher tbh