Well actually i got Qwen3.8-Flash-Next to write one cause i cant read or write anymore, its all gone.........
Strata (the Qwen3.8-Flash-Next inference engine) has a ton of benchmark posts and a ton of "can I run this on my specs?" questions — but no beginner guide. So here's the last three weeks of post condensed, don't waste your time reading it dummy just ask for the summary cause that's all we do now.
Don't moan if you find mistakes, cause i didn't read it all as well !
Where everything in this came from: (r/LocalLLM, r/LocalLLaMA, r/Qwen_AI and friends, last three weeks), the official repo docs (README, INSTALL, MODELS, HOW_IT_WORKS, COMMUNITY_BENCHMARKS, TROUBLESHOOTING), the GitHub release notes up to v0.1.44.1, the big HN thread, the Bodega One walkthrough, and Artificial Analysis for independent model scores. Plus my own benchmarks on my machine (RTX 5090, 128 GB RAM) — every number in it is tagged so you know whether it's an official claim, a community report, or a measured test.
What's in it: what Strata actually is, what to run for your RAM, install-to-first-answer steps, the settings that matter, hardware tables, its rough edges (the quant-quality argument included, not hidden), and an annotated reading list instead of a link dump.
(Skip to the section you need: 1) what it is 2) what it does 3) install 4) settings 5) performance 6) rough edges 7) reading list 8) verdict)
1. Strata in five minutes
Strata runs one very large AI model — Qwen3.8-Flash-Next — on an ordinary gaming PC. The model normally needs 100+ GB of VRAM; a gaming PC has 12–32. Strata's trick: spread the model across GPU + system RAM + CPU + SSD working in parallel instead of forcing it all onto the graphics card. Result: a smart, private assistant, no subscription, no internet, nothing leaves your machine.
What's different about the approach:
- The model is a mixture of experts (MoE): 125 billion parameters, but only ~6 billion activate per word — 10 experts get consulted per word out of 24,576. Knowledge of a huge model, compute cost of a small one. [Official]
- Adaptive expert cache: the GPU keeps a few thousand of the most-used experts and learns which ones matter while you chat; RAM holds all of them; the CPU computes the overflow at the same time as the GPU. SSD holds a 29 GB lookup table, read a few rows per word. [Official]
- Guess-then-check (speculative decoding): a small built-in drafter guesses the next few words; the big model checks them in one pass. Same answers, 1.6–1.8× sooner. [Official]
- Most tools (llama.cpp, LM Studio) are general-purpose. Strata is purpose-built for this one model family. The official same-GPU comparison: 2.5–3.5× writing speed and 5.5–7× prompt-reading speed vs llama.cpp on the same RTX 5090, because llama.cpp shuttles expert data across PCIe every word. [Official]
Who it's for: anyone with a gaming-class PC (12 GB+ GPU, ideally 64 GB RAM) who wants a capable offline assistant or a local brain for coding agents instead of per-token API bills.
Where it stands today: usable with caveats. v0.1.x, ferocious development pace (v0.1.40 → v0.1.44.1 in about two weeks, almost every release with real speed gains). 936 points on the 4 Oct HN thread. But: essentially one maintainer, occasional regression releases (0.1.44 broke older CPUs/non-M4 Macs, fixed in <24 h), and it runs essentially one model family. Excellent, fast-moving beta software. [Atlas rec]
2. What can you actually do with it?
Available now (0.1.44.1) [Official]
- Browser chat at http://127.0.0.1:8080 + a live Monitor (GPU/CPU/RAM, tok/s).
- Standard API: OpenAI-compatible
/v1, Anthropic-style /v1/messages (Claude Code), /v1/responses (Codex CLI). Any chat app or coding agent can plug in.
- Vision (images) optional.
- Long documents: native 262K-token context, prompt reads at 1,600–8,500+ tok/s depending on hardware. My test: 128K-token codebase read in ~15 s on the 5090. [Measured]
- Multi-GPU, LAN serving with API key, MCP server so AI tools can start/stop Strata.
- Model variants: Coder (32 GB RAM), Swift 1.5 (thinks 63% less, ~1.8× faster), Unsloth 4–8-bit files, NVFP4 packs (RTX 50-series), and since 0.1.44 Qwen3.6-35B-A3B too.
BENCH.bat = 5–10 minute self-benchmark that writes a report in the repo's community format.
Planned/speculative [Official]: image generation (endpoint in preview, off by default), speed-projection tool (disabled), Intel Arc + Strix Halo (community, build-from-source), >262K context (community reports ~500K; not supported).
Use Strata when: you have the RAM/VRAM and want the biggest open model your box can hold, you work with huge contexts (whole codebases — reading is where its lead is largest), or you want a local coding agent running unattended.
Use something else when:
- A model that fully fits your VRAM is simpler and often faster. A 30–35B MoE (Qwen3.6-35B-A3B) does 270–380 tok/s in plain llama.cpp at near-identical quality on a community 5-test average (93.3 vs 93.9). [Community]
- Multiple users / parallel agents: Strata is one-request-at-a-time by default. HyperQwen (vLLM fork, dense 27B) hits ~380 tok/s across 8 users. [Community]
- <32 GB RAM or <12 GB VRAM: run a 27B at Q4 in LM Studio instead.
- You want the best answers period: quantised local ≠ frontier cloud.
Is the model actually smart? Artificial Analysis: 39.8 Intelligence Index, rank 6/117 in size class, ~2.2× the median comparable. [Verified tracker] Vendor coding numbers: SWE-bench Pro 62.5. [Vendor] Honest weak spots: those are full-precision scores — you'll run a reduced quant — and the same tracker logs 45.3% hallucination on factual recall. It reasons well, invents specifics. Verify facts. [Tracker]
3. Getting started
Prerequisites [Official]
| Component |
Minimum |
Recommended |
| GPU |
RTX 20/30/40/50 or listed AMD RX 6800–9070/AI PRO R9700, 12 GB VRAM |
16–32 GB VRAM (each extra GB ≈ +700 resident experts) |
| RAM |
32 GB (Coder model) |
64 GB = all standard sizes; 96–128 GB unlocks 4-bit sizes |
| Disk |
~80 GB free |
NVMe strongly preferred |
| OS |
Win 10/11 or Linux, current driver |
Native, not WSL (WSL slower, disables KV RAM-streaming) |
RAM decides which size you can run — check RAM first. Macs exist (Metal) but are pioneers, not the target audience.
The simplest beginner setup: 64 GB RAM + any 12–16 GB card → installer recommends IQ2_XS → press Enter through everything. The defaults are deliberate.
Step by step:
- Close big apps; check free RAM (you want nearly all of it).
- Download the repo (zip or
git clone).
- Windows:
START-HERE.bat. Linux: ./setup.sh.
- Answer the prompts — Enter = recommended each time.
- It downloads the model (~70 GB; if the download drops, rerun — it resumes).
- First start loads 35–55 GB into RAM: your PC looks frozen for 1–3 minutes. This is normal. Wait.
- Browser opens at :8080 → type a question → answer appears → it works.
- Monitor tab: check tok/s against the section-5 tables.
- Optional:
BENCH.bat to baseline your machine.
- Hook up apps: point them at
http://127.0.0.1:8080/v1 as OpenAI-compatible. LAN? --host 0.0.0.0 always with --api-key.
Common beginner mistakes [Official docs + Community]:
- Size bigger than your RAM → endless disk churn, 7–8 tok/s crawl. Symptoms = pick smaller or close programs.
- Panicking at the first load and restarting it. Wait 10 minutes.
- Running from Task Scheduler defaults → 13–14 min startup instead of ~35 s (Windows throttling; documented overrides exist).
- Running under WSL for "speed" — it's slower and disables a long-context feature.
- Thinking "port 8080 in use" is an error — Strata is already running.
- LAN without an API key = an open model for the whole network.
- Comparing other people's tok/s without matching quant + context (headline numbers are always the smallest quant).
4. Settings worth understanding
| Setting |
What it does |
Effect |
| Quant (Q2_0 → IQ2_XS → IQ3_XXS → IQ3_S → 4-bit) |
How hard the model is squished |
Biggest lever. Each step ≈ +15–21% speed, less RAM. Official agreement-with-full-precision: IQ3_XXS 84.5%, IQ3_S 87.7%, 4-bit 88.9–93.1% [Official] |
| Context length |
Working memory |
~1.4 GB per 128K tokens of KV cache at defaults; 64 GB boxes cap big sizes at 128K |
| Reasoning effort (off/low/med/high) |
How long it "thinks" |
Biggest everyday lever. Official puzzle test: off = answered in 6 s; high = blew the cap with 5,600 chars of reasoning and no answer [Official] |
--batch N |
Parallel requests |
Costs speculative-decode boost (11–24% slower per answer on 12 GB cards). One user? Leave off |
--kv format |
Conversation-cache memory |
int8 = safe. k8v4: −23% memory → measured 85→99 tok/s on a 3090 at 198K ctx [Community]; q4_0 slightly faster but measurably worse long-doc scores [Official] |
--kv-resident |
Above ~64K tokens keep cache in RAM |
+23% measured on big contexts; not under WSL [Official] |
--mtp |
Speculative drafter |
Keep on unless ≤8 GB VRAM (dropping it frees ~1 GB for experts — can win on tiny cards) |
--calibrate |
Auto-tunes expert-split, drafter threshold, CPU threads |
Run once per PC+model; keeps only >3% wins. Beats hand-tuning [Official] |
Leave at defaults: KV format, spec decode, parallel slots, prompt cache. Drop reasoning effort to low/off for agent/tool steps; high is fine for chat.
5. Hardware and performance
Rules: "writes" = answer speed, "reads" = prompt ingestion — ~20× apart, never comparable. Match GPU + quant + context before comparing anyone's numbers. [Atlas rec]
Table A — official figures (engine 0.1.26–0.1.36) [Official]
| Machine |
Q2_0 |
IQ2_XS |
IQ3_XXS |
IQ3_S |
writes/reads tok/s |
| RTX 5070 12 GB, R5 7600, 64 GB |
94/2,650 |
79/2,090 |
62/1,750 |
53/1,620 |
Coder 55/2,180 |
| RX 9070 XT 16 GB, 3900X, 47 GB |
60/1,160 |
52/1,110 |
— |
— |
Coder 44/1,420 |
Table B — measured on my box: RTX 5090 32 GB + 128 GB RAM, 14–15 Oct [Measured]
| Size |
Writes (median, short → 128K ctx) |
Reads 128K prompt |
Peak RAM |
| NVFP4 GPTQ 4-bit |
146.5 → 152.5 |
8,535 tok/s (~15 s) |
89 GB |
| Unsloth UD-Q4_K_XL |
132.7 → 138.8 |
6,255 tok/s (~21 s) |
96 GB |
3 runs per point; quality screen effectively tied (34/42 both). Not a general-intelligence proof.
Table C — community reports (user-run, not controlled) [Community]
- RTX 3060 12 GB + 64 GB: usable at all four sizes, recall + HumanEval checks pass (official repo report)
- RX 9070 16 GB + 5800X3D: 57.9–58.7 tok/s at 4K–128K, needles 6/6
- Strix Halo laptop (no GPU, 128 GB unified): 42–59 tok/s, 15/15 needles at 70 W
- 2× V100 16 GB on X99: 30–40 tok/s, ~1,000 prefill
- 2× Tesla P40 (2016 cards): 35 tok/s — ~$200 in cards
- 4× modded 3080 20 GB: 105 tok/s, 150K context
- 5070 Ti 16 GB + 64 GB, week of DevOps: ~half the wall-clock of a full-precision API model;
--calibrate +25% decode
- "$1000 Local AI Build with Strata" (11 Oct): the closest Reddit gets to a setup guide
- 8 GB laptop / 12 GB+32 GB reports: community keeps pushing under the official floor, usable but with caveats
Recurring caveats: dual-GPU card order matters on some boards; spec-decode fails on some multi-GPU/RAM-light builds; sub-4-bit quants "overthink" simple problems per several testers.
Min vs recommended: minimum 12 GB GPU + 32 GB RAM (Coder/Q2_0). Sweet spot 12–16 GB GPU + 64 GB RAM → 50–80 tok/s. Comfort 24–32 GB GPU + 96–128 GB RAM → 4-bit sizes fully resident, 100–160+ tok/s. More VRAM beats a faster CPU — same quant does ~62 tok/s on 12 GB vs ~105 on 24 GB. Windows vs Linux on NVIDIA: no real difference (the "Windows 1.5× slower" test was retracted by its author as a compositor artifact). [Community, retracted]
6. Limitations and rough edges
- One model family (plus Qwen3.6-35B since 0.1.44). Want others → llama.cpp. [Official]
- One request at a time by default; batch slots cost speed on small cards. [Official]
- Very long first prompt = full read every session start (~1 min per 30K tokens on modest hardware). Follow-ups cached. [Official]
- UD-Q4_K_XL crawls on 64 GB RAM (7–8.5 tok/s) — 4-bit files need ~96 GB+. [Official]
- AMD Windows vision is CPU-only and benchmarked less accurate than llama.cpp with same weights. [Community]
- NVFP4 = NVIDIA-only, really wants RTX 50-series tensor cores. [Official]
- No macOS main path; WSL underperforms. 262K+ context unsupported on consumer configs.
- The quant-quality argument, honestly: the dev's own test shows even IQ3_S agrees with full precision only 87.7% of positions; some users report IQ2/Swift quants wandering off-topic mid-reasoning [Community]; another ran IQ2_XS for a week of DevOps and called it near-parity [Community]. Bigger RAM = less compromise. [Atlas rec]
- Verbosity: Flash-Next generated 240M tokens in its own eval run (140M median for peers) — part of its measured intelligence comes from thinking out loud, which is also why reasoning-effort swings latency so much. [Tracker]
- Unproven: multi-year maintenance (bus-factor ~1, though community forks exist), non-technical-family usability, how it ages against whatever Qwen ships next. [Atlas rec]
7. The best resources (annotated, not a dump)
- Best beginner overview: the project README — plain-English explanation, real speed tables for two gaming PCs, which size for your RAM. Unusually honest (publishes its own numbers with engine versions). github.com/Niko1221/Strata [Official]
- Best setup guide: docs/INSTALL.md — every flag, update path, Docker, multi-GPU, older CPUs. There is no better independent step-by-step than the repo's own; AI users get a paste-ready docs/AI_SETUP.md. [Official]
- Best third-party walkthrough: "How to run Qwen3.8-Flash-Next on a gaming PC with Strata" — Bodega One, calm 8-minute guide with the RAM-first sizing table and caveats up front. They sell a competing tool, so not neutral — they say so themselves. bodegaone.ai [Community]
- Best settings explanation: docs/MODELS.md + docs/DETAILS.md — every selectable size vs RAM, then every measurement with engine version. [Official]
- Best hardware comparisons: docs/COMMUNITY_BENCHMARKS.md + the bench/results folder — structured community reports (GPU/CPU/RAM/driver/engine per entry) from RTX 3060 to 5090, AMD, dual P40/V100, Strix Halo. BENCH.bat writes new ones in the same format. [Official collection]
- Best technical explanation: docs/HOW_IT_WORKS.md (+ the paper in docs/paper/) — two pages, kitchen analogy first, then the precise GPU/RAM/SSD division of labour. Best explanation anywhere of why a MoE runs on a gaming PC at all. [Official]
- Best troubleshooting: docs/TROUBLESHOOTING.md + GitHub issues — covers the four things that actually break; the maintainer ships fixes credited to reporters within days. [Official]
- Best place to track updates: the Releases page — ~weekly, each note is a mini-benchmark with "defaults that changed" sections. [Official]
- Best community discussion (skeptic included): the HN thread, 936 points, 420+ comments — owner benchmarks, quant scepticism, practical tips. news.ycombinator.com/item?id=49953495 [Community]
- Best ongoing Reddit signal: r/LocalLLM search "strata" — densest cluster of real-hardware reports; read alongside "Flash Next vs 27B — The hype vs the truth" (13 Oct) for the reality check. [Community]
- Category with no good resource yet: a neutral, controlled quant-quality comparison against the full model. The agreement percentages above are the developer's own test (rerunnable: tools/quant_quality.py). Nobody independent has published one. Until then, weigh quant-quality claims accordingly.
8. Bottom line: should a beginner try it now?
Yes — if your hardware clears the bar. With 64 GB RAM + 12 GB+ card, Strata is the only thing that runs this class of model on a gaming PC, privately, at reading speed, and the default path is literally Enter ×4. The caveats are mostly about the model (quantised quality, verify facts, one user at a time) more than the software. [Atlas rec]
No — if you're under 12 GB VRAM / 32 GB RAM (27B in LM Studio instead), you need parallel users (vLLM/HyperQwen), you want a maintained product rather than fast-moving OSS, or "best answers" beats "local and private" (cloud still wins). [Atlas rec]
Start-here checklist:
- ≥12 GB VRAM, ≥32 GB RAM (64 ideal), ~80 GB SSD.
- Pick by RAM: 32 → Coder · 64 → IQ2_XS (or IQ3_XXS if patient) · 96+ → IQ3_S / 4-bit.
START-HERE.bat / ./setup.sh, Enter through defaults.- Wait out the 1–3 min first load.
- Test at :8080, check Monitor against Table A.
--calibrate once, BENCH.bat once.- Only then wire up apps at
/v1; drop reasoning to low/off for agents.
Read first: ① the README (10 min, answers 80%) ② the HN thread (scepticism + real numbers) ③ HOW_IT_WORKS.md (makes everything else obvious).
Every factual claim is tagged [Official]/[Measured]/[Community]/[Vendor]/[Atlas rec]; my measured numbers are RTX 5090 32 GB + 128 GB RAM, engine versions named, sources checked 16 Oct 2026.
