I have a 3rd gen Threadripper paired with this board, which annoyingly has only three PCIe slots.

https://rog.asus.com/motherboards/rog-strix/rog-strix-trx40-e-gaming-model/

I am currently running 3x 3090's at PCIe 4.0 x16 each and I get around 100 t/s using QFN on Strata. Now I wish to add a 4th 3090 to run other models and I am not sure whether to use bifurcation or upgrade the motherboard, or the entire system (mobo+cpu).

I had a look at the c-payne SlimSAS cards to bifurcate to x8/x8 but once shipping costs are considered these tend to work out to be rather expensive. There are some Chinese counterparts on Amazon, Alibaba, etc. but I am unsure about how reliable these are.

The other concern I have is whether using x8 will lower performance on Strata since it seems to be more reliant on the system as a whole and not just the gpu's. Would appreciate some advice on how to add the 4th card.

Yesterday Fish released this incredible text-to-speech model, so naturally I spent the rest of the day building a great demo for them :D The model is a chunky MoE (35B3A), but it runs well on a DGX Spark. The 4bit quants get very close to the original. You can try it here: https://huggingface.co/spaces/fishaudio/Drama-3 or download the model and run it locally 🤗

I'm planning on maxxing out my pcie slots with rtx 6000 blackwell cards for tp4. I'm wondering if its possible to use an egpu dock like the razer v2 and connect it via thunderbolt to usb 4 on the back of my mobo. The egpu in question would be another blackwell card (5070ti), and its sole purpose is for sidecar models, and to act as a display card for my rig so that i dont have to go headless.

Do you guys know if this is possible?

I read this thread recently and it kind of scared me off:
https://www.reddit.com/r/LocalLLaMA/comments/1vlfe9f/egpu_folks_rtx_pro_6000_blackwell_egpu_crashes/

Step 5 Preview is StepFun's flagship model for agentic work. It delivers frontier-level performance across software engineering and professional knowledge work, with particular strength in finance.

Saw soneone post their bathroom AI PC a few weeks ago. Figured I’d contribute with my chicken-coop adjacent shed AI PC.

Started with a single CMP 170HX (unlocked to 64GB) and ended up with three more for a total of 256gb VRAM. Found a deal on a used Threadripper with a TRX40 motherboard and 256gb DDR4 RAM to host the 4x 170HX. Added a 3050 plugged into riser connected to a NVME slot for video/RDP. My case ran out of space so that’s the board dangling freely lol.

My original goal was to save money by running AI locally. Naturally, this escalated into four 170hx, a 32 core Threadripper, and enough RAM/VRAM to turn my hobby into what my wife says is a financial liability… oof.

Originally had it in a closet, but between the heat and the wife complaining about the noise, the GPUs were eventually banished to the shed. Now they are in a happy place with A/C and 2x 20amp receptacles on separate circuits. One for each power supply. No more fan curves to reduce noise either.

Running QFN split between two of the 170HX with capacitor mods and Qwen3.8-27B on the third. Just got the 4th 170hx installed today. Downloading GLM 5.3 Flash EXL3 4.65 bpw to see how it’ll run on all 4.

Really looking forward to seeing how this beast will handle Qwen4. QFN does so much so well that I can’t imagine what Qwen 4 will be like.

Looking for people who have tried this model and was able to use it on real-world tasks. It's what Strata list as "coder". What was your experience like?

On my setup I am not really sure if it is worth more than iq3

Hey everyone, I’m trying to decide what to do with my local LLM setup before putting more money into it.

A bit of background: I’m a final-year electrical engineering student, and I use local models for learning about LLMs, experimenting stuff and my thesis on financial analysis of solar plants with BESS. I also am going to work in strategic consulting after graduation, so I want a setup that is useful for data and market analysis and some coding.

My current machine:

• i7-7800X
• MSI X299 Raider
• 128GB RAM — 8×16GB DDR4 2666MHz
• 2× RTX 3090

I’m currently running Strata IQ3_S at around 100 tokens/s, and I’m pretty happy with the speed. Quality seems okay, a bit better than 27B at Q8.

I also have five more 3090s: one ASUS ROG Strix and four Zotacs. All five need repasting and new thermal pads. So seven cards total, with only two currently in use.

I paid about $900 each for the four Zotacs. That’s part of why I’m torn: I could use them for a bigger setup, but I could also sell them for around $1,300 each pretty easily in my local used market.

These are the options I’m considering:

1. Upgrade the existing X299 machine

I found an i9-10940X on AliExpress for around $300. It would give me more cores for my other server workloads and more PCIe lanes, without replacing the motherboard and RAM.

My understanding is that I could then run the GPUs at x16/x16, depending on the motherboard’s slot wiring. Please correct me if there’s a limitation I’m overlooking.

Alternatively, I could use more widely spaced slots at x16/x8. I’ve tested that layout, and the top GPU runs about 10°C cooler, with the cards sitting around 65°C and 58°C.

Would you spend $300 on this CPU upgrade, or avoid putting more money into X299? For inference, would you prioritize x16/x16 or the better cooling at x16/x8?

2. Build a separate machine for the four Zotacs

This would give me a second machine with 96GB of VRAM. I’d need to buy a suitable CPU/motherboard setup and sort out cooling, power, and the rest of the build.

For anyone who has gone from 48GB to 96GB:

• Was the improvement in model quality worth it? Im looking at maya for GLM 5.3 flash.
• Which models and quantizations made the upgrade worthwhile?
• What inference speeds are you getting, and with which backend?
• What reasonably priced platform would you recommend for four 3090s? Used hardware is fine.

I’m happy with my current speed. The main reason to expand would be access to better models or higher-quality quantizations for engineering, financial modeling, document analysis, and coding.

3. Sell the Zotacs and keep the current setup

Since I paid about $900 per card and could sell them for around $1,300 each, I could also sell them and avoid the cost and work of another build.

Would you keep the cards because 96GB opens up enough useful possibilities, or sell them while the local resale price is still decent?

Just downloaded the new LFm2.5-24B-A2B LiquidAI model for testing. They claimed some pretty good performance, and I've had good experiences with their prior releases.

Some context - I use Ornith 1.5-35B pretty regularly in multiple workflows - sometimes on it's own, and sometimes as part of an AI-Council in Pi Agent. It's been very reliable, provides surprising insight "smarter" models miss, and is a great workhorse in Pi.

I swapped LFM 24B in to replace Ornith in a few of my workflows - it is holding up REALLY well. It's very fast (~95 tps sustained in llama.cpp using ROCM on an AMD W7900 - latest llama.cpp build). The Q8 takes up 30GB vram with full 128k context. It hasn't failed me yet in my mechanical tasks - it just keeps going and going and going and seems to tick all the boxes for a "little" model I run alongside a larger or smarter model on my other GPU.

For "insight" I'll have to test it longer in my council-mode workflow but I will report back.

Just wanted to give my $0.02 (CAD) on this model - it is very competent with Pi and worth your time to check out. The speed alone is a nice upgrade, and for my more basic tasks, even long running recursive tasks, it is holding up very nicely.

Kudos to the LiquidAI team, this one is going to get alot of use on my chonky boi.

For anyone interested, I've been doing some kernel and tuning optimization for AMD because it was performing abysmal out of the box. But the effort was worth it as I was able to more than 2x its performance out of the box (the card is extremely new and there are almost no GEMM table and supports yet). (due to both using separate models and environmental factors input token can have ~3% less tokens for AMD to decode despite effort to keep them as close as possible). Just to keep things short and concise, I did a varying tests of small input c=8, medium input c=20, and long input c=1. Everything was done in cold state (no kv-cache at start).

And I think there is more performance to unlock for MI350P as below benchmark was performed when I was only able to unlock 50% mem utilization (currently inference is still heavily bounded by compute because of missing GEMM tuning config)

Model used:
AMD: amd/Qwen3.8-27B-Quark-AWQ-MXFP4
Nvidia: unsloth/Qwen3.8-27B-NVFP4

(Small) 3k-input, 8 user comparison, 100 prompts
AMD:
Total input tokens: 288653
Total generated tokens: 75000
Nvidia:
Total input tokens: 300000
Total generated tokens: 75000

(Medium) 50k-input, 20 user comparison, 20 prompts

AMD:
Total input tokens: 965153
Total generated tokens: 10000
Nvidia:
Total input tokens: 1000000
Total generated tokens: 10000

(Large) 200k-input, 1 user comparison, 5 prompts
AMD
Total input tokens: 962033
Total generated tokens: 25000
NVIDIA
Total input tokens: 2000000
Total generated tokens: 25000

I saw the 4x R9700 build for a 10-person startup here a few days ago and wanted to share my results, since I opted to spend most of the money on the cards. I went with a super cheap used EPYC instead of a Threadripper. In the end it came to about $10.7k per box all-in, and I built three of them for a total of 384 GB of VRAM.

As a long time lurker on my other account, I'm always frustrated that people never show how they actually use the models, so after a lot of back and forth I decided to share the actual project they power instead of being vague.

I use these to run the AI behind my business, USA Arms. We organize tens of thousands of firearms-related businesses (gun dealers, ammo dealers, ranges, etc.) and that info is always changing, so we need a system that keeps the site up to date. The models check source facts, sort out business identities, and pull out stuff like services and hours. Every answer gets verified against what the agent actually saw, and we store the documentation so bigger models can audit the work.

I didn't plan to move the majority of my work to local AI until later this year, but hosted models started refusing to do normal research for my business ~2-3 weeks ago. They refused to identify a gun dealer because the license holder might be an individual, and wouldn't look at an ammo dealer's website because they can't help anyone buy ammo. Our harness just started breaking in random ways, so I'm glad I made the leap.

We tested Luna 6, DeepSeek, GLM 5.3, Qwen 3.8 27B, Gemma and Inkling on our workload, along with Kimi, GLM Flash and others. GLM gave us the best results. Qwen gave us the best cost-to-performance, so that's what these run.

Enough ado, into the numbers:

Per box, $10,733.91 with tax and shipping (averaged over the three)

• 4x ASRock Radeon AI PRO R9700 Creator 32 GB (128 GB VRAM), each capped at 210 W: $7,244.58
• AMD EPYC 7452, 32 cores, used: $127.17
• ASRock Rack ROMED8-2T/BCM: $1,162.39 (there was a global shortage, ours came from Helsinki)
• 8x 16 GB DDR4-2666 ECC RDIMM, all eight channels: $860.59
• SilverStone HELA 2050R Platinum: $578.92
• SilverStone RM44 4U: $388.40
• Samsung 990 PRO 1 TB: $254.39
• ARCTIC Freezer 4U-M, 3x ARCTIC P12 Pro front, 2x Noctua NF-A8 rear, SP3 frame, paste: $117.48

The GPUs are 2/3 of the cost, but the boxes also run our CI/CD, which had become a bottleneck in my businesses. I hadn't seen anyone discuss this: they run Rust builds just fine if you tune them to prioritize inference. In our test, running builds alongside inference didn't measurably change throughput, and the builds were only 3-8% slower.

Engine

• vLLM-Radiance 1.0.387 (vLLM 0.30.0, ROCm 7.14). Huge thanks to the Radiance folks; their R9700 work is what we run.
• Qwen/Qwen3.8-27B-FP8 (Qwen's official FP8 checkpoint), FP8 KV cache, the model's own MTP head with 8 draft tokens
• Two engines per box, TP=2 each, 8 sequences per engine, 256K max context, about 488K tokens of KV cache per engine

Results, vllm bench serve, one box, 2x TP2:

• 1 user, MT-Bench: 53.1 tok/s, TTFT p50 125 ms
• 16 users, MT-Bench: 484 tok/s aggregate, about 40 tok/s per user
• 16 users, random 1024 in / 512 out: 623 tok/s aggregate, 837 W average for the four cards
• 32 users (a bench profile with max-num-seqs 32): 603 tok/s

MTP depends a lot on the text. On our JSON extraction, 83% of drafts get accepted (7.6 tokens a step), vs 39% on MT-Bench (2.7 a step), so MT-Bench undersells it for structured output.

Long prompts, 8 unique real-text prompts at once (4 per engine), 128 out:

Every prompt was unique (0% prefix cache hits).

FP8 vs MXFP4 on a real workload

The startup build ran MXFP4, so I had AI run that too (Radiance's fastest profile) on one box against our own graded extraction workbench:

• FP8: 98.0/100 correct, 4,477 validated jobs an hour
• MXFP4: 95.3/100, 4,819 an hour
• On a holdout set: FP8 94.0, MXFP4 90.0

So FP8 roughly halves the wrong answers for 2–7% less validated throughput. Sticking with FP8 was an easy call for this workload, where "false facts" that make it into the system are a real headache.

For reference, our six Mac Studios (four M4 Max 128 GB, two M3 Ultra 512 GB) on MLX 8-bit scored 99/100, but at 723 an hour across all six. Plan A was buying more Macs, and keeping up would have taken at least 41 more, about $280k.

I made a video of the build targeted more at beginners and people trying to move AI locally: https://youtu.be/o7k5Kj-ho-M

Happy to answer questions. The first of which I expect to be how the heck were the Macs so slow, and I'm sure I'll get flamed haha.

Inspired by how short-term memory works in our own brain, I built PulsarLM, an inference engine that removes old and unimportant tokens from the context window to keep it bounded while also allowing the conversation to go way past the trained limits.

Running Qwen3 8b on a 5090, I get 6.7k tok/s during prefill and 89 tok/s during decode. It does support batched decode and with 16 concurrent requests, I get 1.2k tok/s.

It works by using custom attention kernels that accumulates an attention mass per token page. At regular intervals, pages that receive low attention mass gets removed.

To workaround out-of-distribution positions, embeddings are re-rotated (for RoPE) such that token positions stay contiguous and never exceed the context limit.

Eviction of tokens in the context window would also break prefix caches normally, but PulsarLM also caches snapshots of context to avoid unnecessary re-prefills, but does use up extra RAM. There is also an option to persist those snapshots.

Only a smaller and older models are implemented for now, and the engine itself still has rough edges. It's only been tested on linux x86_64. Feedbacks are welcome! Also, what models would you be interested to try?

Here's the link: https://github.com/cosmonomic/pulsarlm

Edit: There used to be a recall mechanism using bm25+attention score, but removed it since it didn't make sense to store that many KVs. So right now, it works best when paired with an external memory system/RAG.

And also, running benchmarks right now, will update once results are out.

UserRepGen-4B is a fine-tuned Qwen3.5-4B language model that generates a structured, natural-language user representation from a multi-turn user-assistant conversation. Each representation contains a task description and a behavior description for a user, and is intended to condition a separate LLM-based user simulator.

Unlike UserLM-8b and UserLM-2.0, which generate user utterances, UserRepGen-4B generates representations, not simulated user turns. It does not itself execute the user's task or serve as the simulator.

The model was trained with online supervised fine-tuning through hindsight distillation, using feedback obtained by comparing simulated and original user behavior. At inference time, it generates a representation directly from the source conversation without requiring simulation rollouts or hindsight feedback.

https://huggingface.co/microsoft/UserRepGen-4B

As strata is somewhat new, I assume that some people din't yet learn how to finetune it for best performance on their machine. I find documentation a bit hard to navigate (use your agents for this!), which doesn't help.

After some serious testing, I've tweaked pretty much every parameter available. Some are obvious and I'm not going to waste your time on explaining mtp tweaking.

Some I beleive are more obscure and those are the ones worth mentioning:

1. Strata setup is supposedly picking the best --pcie-frac value for you. I find that it often doesn't. It's worth investing your time and testing it. Counterintutively, my best performing value is a low 0.2.
2. By testing different --pool-workers values, I was able to find an easy +10% in decode. I'm not smart enough to explain what it does (and how it's different from point 4), but it's worth investing your time in running some benchmarks on your machine.
3. --trim-stage-weights. Boom, extra +8%, try it. Doc is quite explicit on what it does if you're curious. I do beleive that this one can impact performance later in the session and increase swaps but it's still worth it.
4. --pipeline-windows. All I know is that higher values make my CPU work extra. Going from nothing/default to 6 gave an almost 15% increase.
5. Last but not least multi-GPU owners be mindful of GPU priority, if one of your GPUs is on a slower pcie lane. I have two RTX3060's. GPU 0 and GPU 1. At some point I noticed that one of the two GPUs basically acts as a memory pool and doesn't do any compute. I guess it's makes sense for a MOE model. But now I wanted to understand which ones eactly does the work.

So I quickly figured out that my default GPU 0 is the one on the slowest pcie lane! If your have two identical GPUs on different speed lanes, bear in mind that their 0/1 order is basically random. You need to figure out exactly which one is which and force the quick one to do the inference. Ask your agent for command lines to figure out whether the 0 or 1 is on your pci_e1 lane. In my case it was GPU 1, the only way to set it as the main GPU is a not very user-friendly two liner :)

"gpu": [1,0],

"gpu_order": "as_given"

This was like another +15%, ridiculous.

With all of those cumulated, I'm seeing speeds that I should not be having on my machine. I went from something like 35 decode peak on Qwen Flash Next IQ3_S, to 60 tps average code generation on empty context and 60-70 tps peaks.

Performance does degrade with full context but still easily a solid +15%-20% versus the previous baseline tests. This includes huge sessions on full context and compaction.

So, try it out.

Anecdotally, I've noticed that QFN seems more collaborative than GLM. QFN seems willing to engage in conversation. It feels more like an assistant. GLM seems more action-oriented and less willing to converse. That is, I ask GLM a question to engage in discussion: GLM runs off searching the repo and then often re-engages in development without discussion.

I say this using a self-hosted QFN q4_k_xl versus GLM 5.3 Flash hosted on OpenRouter!

I wonder if others have had similar experience? If so, have you found any particular prompting to cause GLM 5.3 Flash to me more interactive and collaborative?

I suppose I could try lifting some prompting out of pi-goal-x or similar as that seems to mostly succeed at forcing GLM 5.3 Flash to the table. However, I found that GLM will often take the information given as part of the process, run off, and implement most of the solution before even negotiating the goals!

Hey everyone,

I’ve been working on a project called OpenSpline, an open-source avatar layer for existing voice agents. You supply a portrait and your agent’s audio, it generates synchronized talking video on your own NVIDIA GPU. The lite model can work locally on a single GPU.

It’s still early, but the results are super promising.

Would love to hear your thoughts, feedback, or ideas on what you'd want to create using it!

Code: github.com/openspline/openspline
Demo: openspline.dev

Hey folks; question for you - are any of you using open weight models running locally for actual development tasks? If so, what models are you using, and what kind of tasks are you giving them?

I ask, because I've made an open source harness (you can play with it, but it's still beta and I have done almost no windows/linux testing and the first run experience is not great yet). A big part of how it works is that there are three routing tiers:

Primary | Secondary | Local.

• Primary - What runs your main agent, and for instance, i use it with my ChatGPT/Claude account.
• Secondary - What the main agent primary delegates to. I personally use a deep infra account and GLM 5.3 + GLM 5.3 flash. GLM 5.3 is slow, but super capable. 5.3 air is fast and nearly as capable.
• Local - The harness routes to this for truly local tasks. Git workflows, documentation, searching through code and creating summaries, etc.

The thing about local, is that I have a 128GB M5 or whatever, and I run llama-server with GLM 4.5 air. And with a proper context size (128K), it takes like 96GB of RAM or something. I have tried Qwen3, Gemma3, and some older deep seek models and anything less than GLM 4.5 air has been pretty disappointing. It's possible that I had Qwen3 or deep seek not properly configured (and I've made some changes since to better handle tool calls and whatnot), but I had very little luck with them. I've seen the benchmarks, and they make me believe that I shouldn't need a 96GB model to get decent local development experience.

So if you've run some local models with good experience, what have you been running? what amount of RAM on your machine do you have? I really want some good out of the box configurations in Cercano for folks that have a 48GB/96GB machine, and your experience can help me here.

I have been working on a side project the past week. I am running a single R9700, Qwen3.6-35B-A3B-UD-Q6_K with a 220k context size. I use the claude cli as my harness and I have been testing out the AWS AIDLC 2.11 workflow.

Stats from Claude :
Sessions: 46 Longest session: 13h 37m 31s

Active days: 5/7 Longest streak: 3 days

Most active day: Oct 14 Current streak: 3 days

Input 20.7m · Output 3.2m · Cache read 1.4b · Cache write 0

A few notes... Qwen 3.60 35B has not been perfect. As context grows, it sometimes get confused with the workflow commands. There are some lost tokens, and multiple turn code reviews, until it got things perfect. Opus would probably have been better there, so the numbers below are probably not exact. On the other hand, my llama cache has no set expiration if I pause work. I have been considering paying for Opus just for code reviews and planning, I have been using ChatGPT $20 plan for that by dragging files and copying and pasting stuff i want a second agent to review, which is annoying but it works.

Estimated costs from chatgpt compared to Opus 5.5 :

$426.80

46 sessions · 5 active days · Standard API pricing

Cost breakdown

Pricing verified against Anthropic's official pricing documentation.

Claude Platform Docs

+1

Session statistics

Average per session

$9.28

Per active day

$85.36

Cache read share

65.6%

Where the $426.80 goes

Cache reads

Input

Output

The interesting part: your cache usage

You've accumulated 1.4 billion cached input tokens. At the standard, uncached Opus 5.5 input rate, those tokens would theoretically cost $5,600 instead of $280.

Your local Qwen setup is therefore processing a workload that would have a meaningful API cost, even with prompt caching.

The $426.80 is an estimate, not a billing statement. It assumes equivalent Opus token counts, standard API rates, and no additional tool or service charges. Your displayed statistics are also rounded.

For context, you're averaging roughly $9.28 in equivalent Opus API usage per session. That is a useful benchmark when evaluating the economics of your local AMD R9700 system and the lighter workflow we're designing.

hi , howdy , how are you ^_^

I'm looking to stop using apis and generally after being a long time user of both Janitor AI and Character.ai , I'd like to go fully local. is their any suggestions you have for a good ai waifu model under these specs

DEEPCOOL GAMMAXX GTE V2 Windows 10 Home Plus Team T-FORCE VULCAN Z 16GB (2 x 8GB) 288-Pin DDR4 SDRAM DDR4 3200 Western Digital WD BLUE SN550 NVMe M.2 1TB INTEL I5-10400F Gigabyte B560 DS3H AC-Y1-B EVGA 500 BR, 80+ BRONZE 500W Gigabyte RTX 3060 Ti EAGLE OC 8G (rev 2.0) H510 Compact Mid Tower White/Black

Just built a new PC, full parts list is here. I am using Deadcode's Radiance inference engine, optimized for 2xR9700 and his Q4. The throughput benchmark tool is BetterBench.

Gold Rush of advanced coding AI available to the public will END in a few months after a massive legislative crackdown justified by the need to "save economies and prevent terrorism". Advanced coding will only be available to corporations and governments.

.

Local AI will remain a gray zone as it requires massive amounts of hardware to even remotely resemble frontier capabilities (no matter what the enthusiastic posts on this reddit claim).

.

CIA will put people accumulating enough AI-related hardware to achieve "frontier-like capacity" on lists, so it will be easier to pinpoint where the next "Photoshop clone" will appear.

.

The Photoshop clones, et al, that we have now, will become illegal and will be erased under new legislation, and people making them will be legally pursued.

.

UNCENSORED local AI will become purged from HuggingFace and will be relegated to torrents. Not because it's powerful, but because of the above "terrorism" justification will put it under its umbrella.

Thank you to jesdga95, the creator of Basalt, a fork of Strata for Blackwell hardware! I just got my 5060ti16gb and my measly 32 GB of ram system to 75tok/sec and 1200 prefill at 128k(q2 model ofc) . I had to go q4 on the kv to do it, but I can accept that. This is totally awesome. my system is hanging on for dear life with like 5gb left for the OS and all programs, but as a dedicated coding agent, I could not be happier, considering what I have... Whats even cooler is that he literally reached out to me when I was complaining about strata q1 and was early on my quest to get 3.8 Flash next working well. it is so cool to see an alive internet community where people are literally building the tools and helping others with them directly. Very kind.

https://github.com/jesdga95/basalt

I had to have deepseek4 do shenanigans and add a build flag to allocate more ram, but it worked!! Thanks so much :)

Hey folks,

We all know that reading walls of text sucks. Many labs/companies have been moving towards bringing AGUI into the existing chat window, but that always felt broken. I mean I've never seen Tony Stark write a prompt :)

It is what you want It to be.

Agents already know the best way to get the information from you for a particular task, It gives them a space to do so.

The 2-way communication loop's execution is so good that agents can form full-on dynamic workspaces for each specific task: study math together on a collaborative whiteboard, dynamic live dashboards, etc.

The sick part is that it works with any agent harness you may already be using!

Just tell your agent:
"Install itcan.do"

or install it yourself at itcan.do

GitHub: https://github.com/Aaryan-Kapoor/it

May your GPUs stay warm(or cold?)!

tl;dr: new qwen finetune that simultaneously pushes the frontiers of small model capability and reasoning efficiency.

Hey everyone,

We publish two post-trained versions of Qwen 3.8 27B called Sabi-Noema and Sabi-Noema-MLA. These models push the capability frontier on <30B models with 2x reduction in verbosity, and are exclusively trained on open source environments.

These models have improved capabilities on coding, agentic and tool calling tasks, as well as hallucinating lesser. Sabi-Noema beats GPT 6 Luna-medium on DeepSWE, placing in between Luna 6 medium and high. It beats Luna-high on agentic and tool calling benchmarks like TB2.1, Tau-3 bench, ClawEval, and BFCL v4. Its hallucination rate is considerably lower than Luna, Qwen and other competing models.

These models are much more reasoning efficient. Sabi-Noema generates 50% fewer tokens per response on TB2.1 and 25% fewer tokens on DeepSWE, compared to base Qwen. Sabi-Noema-MLA goes even further by doing a post-hoc MLA conversion of Qwen's GQA attention. This conversion uses only a calibration dataset and no extra training. With BF16 latents Sabi-Noema-MLA needs 3.6x lower bytes per token. The MLA latents can be further compressed to FP8 with the total KV cache size on TB2.1 to 8.4x smaller with similar performance. To put this in perspective, when using Sabi-Noema-MLA at q4_0, on a 24GB GPU like 3090/4090 you can get a context size of ~534K where base Qwen at FP8 cache would only get to ~150K context.

We are excited to have the open source community play around with these models. We will be releasing more post-trains in the future focused on making <30B models more powerful.

HuggingFace:
Sabi-Noema: https://huggingface.co/sabicap/sabi-noema
Sabi-Noema-MLA: https://huggingface.co/sabicap/sabi-noema-mla

I have 32GB of RAM.

MacStudio 128GB M1 ultra

I'm curious how people who can run both models are deciding which one to work with.

DeepSeekv4 benchmarks right below GPT 5.5 high.
Flash Next benchmarks right above 5.4/5.5 xhigh

Both seem capable but it seems like Qwen 3.8 Flash next is taking much longer to actually finish anything. From the chart, it seems like I can do 5 tasks with DeepSeekv4 in the time it takes to do 1 with Qwen 3.8 Flash Next and Flash Next isn't 5x better than DeepSeekv4. In my personal experience it may not be 5x longer but it's absolutely taking signifcantly longer to do anything.

Is there something I'm missing?

NOTE: I'm not bashing Qwen 3.8 flash next, I like it, currently 2h and 5 minutes into deep diving on some work for me. On my hardware I can load one of these models with multiple slots so I have to choose. I'll likely have to invest the time into a more deterministic benchmarking on my personal tasks to compare the quality I'm getting back and I'll still be using 3.8 27b because I find it capable and it uses less of my machine so I can do more other things like the image gen I'm working to integrate into my personal harness.