Strata performance finetuning - try it out

[u/Dreeew84]

As strata is somewhat new, I assume that some people din't yet learn how to finetune it for best performance on their machine. I find documentation a bit hard to navigate (use your agents for this!), which doesn't help.

After some serious testing, I've tweaked pretty much every parameter available. Some are obvious and I'm not going to waste your time on explaining mtp tweaking.

Some I beleive are more obscure and those are the ones worth mentioning:

1. Strata setup is supposedly picking the best --pcie-frac value for you. I find that it often doesn't. It's worth investing your time and testing it. Counterintutively, my best performing value is a low 0.2.
2. By testing different --pool-workers values, I was able to find an easy +10% in decode. I'm not smart enough to explain what it does (and how it's different from point 4), but it's worth investing your time in running some benchmarks on your machine.
3. --trim-stage-weights. Boom, extra +8%, try it. Doc is quite explicit on what it does if you're curious. I do beleive that this one can impact performance later in the session and increase swaps but it's still worth it.
4. --pipeline-windows. All I know is that higher values make my CPU work extra. Going from nothing/default to 6 gave an almost 15% increase.
5. Last but not least multi-GPU owners be mindful of GPU priority, if one of your GPUs is on a slower pcie lane. I have two RTX3060's. GPU 0 and GPU 1. At some point I noticed that one of the two GPUs basically acts as a memory pool and doesn't do any compute. I guess it's makes sense for a MOE model. But now I wanted to understand which ones eactly does the work.

So I quickly figured out that my default GPU 0 is the one on the slowest pcie lane! If your have two identical GPUs on different speed lanes, bear in mind that their 0/1 order is basically random. You need to figure out exactly which one is which and force the quick one to do the inference. Ask your agent for command lines to figure out whether the 0 or 1 is on your pci_e1 lane. In my case it was GPU 1, the only way to set it as the main GPU is a not very user-friendly two liner :)

"gpu": [1,0],

"gpu_order": "as_given"

This was like another +15%, ridiculous.

With all of those cumulated, I'm seeing speeds that I should not be having on my machine. I went from something like 35 decode peak on Qwen Flash Next IQ3_S, to 60 tps average code generation on empty context and 60-70 tps peaks.

Performance does degrade with full context but still easily a solid +15%-20% versus the previous baseline tests. This includes huge sessions on full context and compaction.

So, try it out.

[u/ExcitingLeather5197]

these are the kind of posts i save and then forget about for 3 months

the pcie lane thing is so real though, had a similar issue with my 3060s and it took way too long to figure out which one was actually doing the work

[u/Studio271 → u/ExcitingLeather5197]

Yeah, PCIe lanes matter a lot, I have a 5070ti on a 5.0x16 and a 3060 on a 4.0x4 with 64GB DDR5-6400, and I actually get worse performance when tying to use them both. Currently using the 3060 for my monitors and vision module, but I think even that's probably not as efficient as just letting Strata load/unload it like an expert (an option I think I saw get added recently?).

[u/DistanceSolar1449 → u/ExcitingLeather5197]

You have a codex account? Just give codex the link and tell it to tune strata for you, and run benchmarks. Leave it alone overnight and it’s done.

[u/Bulky-Priority6824]

My problem with Iq3s was terrible nvme thrashing. So much it made the system unusable. Q2_0 works very well. What can I try to smooth out iq3?

I'm only working with 32gb/32gb 

[u/Studio271 → u/Bulky-Priority6824]

more RAM.

[u/Bulky-Priority6824 → u/Studio271]

ok i managed to tune iq3xss config a bit and now its running fairly performative and similar to iq2

[u/AvidCyclist250 → u/Bulky-Priority6824]

Maybe its OS related. Dont have that on cachyos

[u/zovix]

Like you mentioned, I had my subscription brain read through all the docs for Strata, the model, and my system and then offer some suggestions. No fine tuning yet but these were two biggest thing for me so far:

1. Changed --kv cache from q4 to int8 AND enabled env setting STRATA_HIP_WMMA=1.
2. This is apparently specifically for AMD 9060 XT cards and is not done on default. It requires int8 but Strata install recommends q4 so it gets auto disabled.
3. Second was reserving VRAM --vram-reserve-mib = 1536.
4. Now the rest of my computer runs fairly smooth and thus doesn't bog down everything. Previously while I tried to do other things it was lead to huge slow downs.

Over all it is still slower then I would like but my computer is 8 years old so I won't complain to much.
500-800 prefill and 30-50 decode depending on how much context is filled (IQ3_XXS with 262k context window).

[u/darksteelsteed → u/zovix]

With my 16gb 5080 I found that I needed to set the reserve vram to 512. This makes it allocate just under 15gb. Also for nvidia cards tell the driver to not offload cuda to system ram automatically. That fix is needed for all inference engines you run, otherwise the driver lies and claims you have more vram than you actually do leading to awkward token speed drop-off that is really hard to pinpoint. Makes fine-tuning impossible without it

[u/lorde_dingus → u/zovix]

Any insight on your intended fine tuning steps? What hardware are you running?

[u/zovix → u/lorde_dingus]

I have a good 2018 gaming pc! 9060 XT (16GB) and 64GB DDR4, Intel i7-8700k.

My first plan is to backup my config (and PC) then tell Qwen (subscription 🧠) too take the wheel and run benches for a day.

[u/No_Dig_7017]

Nice. Very cool. I'll test these out 

[u/Interpause]

afaik trim stage weights and pipeline windows sre multi gpu specific options according to the docs

[u/royalflash417]

What is your setup and how much RAM capacity you have?

[u/Dreeew84 → u/royalflash417]

2x Rtx3060 12Gb + 64Gb ram