Two months ago we released a fully quantized NVFP4 Qwen3.8-27B trained with QUASAR. We've since trained QUASAR checkpoints for a bunch of other recent models: Qwen3.5-4B, Gemma 4 E4B and 12B, and Muse Glimmer 30B, in NVFP4 for vLLM and Q4_0 GGUF for llama.cpp / Ollama / LM Studio. The code and a project page are now public too.

Every checkpoint is trained with quantization-aware distillation: the original BF16 model is the teacher, and the quantized model is distilled (healed) on the teacher's own responses. The exports are standard formats (so no custom kernels and/or conversion step).

KL to the BF16 model on held-out prompts (lower=better):

On benchmarks:

• Muse Glimmer 30B NVFP4: at or above BF16 on GPQA-Diamond (63.6 vs 62.1), MMLU-Pro (83.3 vs 83.1) and AIME'25 (90.0 vs 86.7), and the only 4-bit build at BF16 performance on RULER at 32K, 64K and 128K.
• Qwen3.5-4B NVFP4: best AIME'25 (74.2) and GPQA-Diamond (77.9) among the 4-bit builds we tested.
• Gemma 4 E4B / 12B: higher than Google's QAT on every downstream benchmark we ran. On H100, E4B also gets +36% batched throughput (group-64 quant uses vLLM's Machete kernel, the faster one).
• Qwen3.8-27B NVFP4: all 496 linears in NVFP4 (W4A4), 90.9 GPQA-Diamond vs 91.4 BF16, at 19.7 GB.

Run it:

vllm serve QUASAR-QAT/Qwen3.5-4B-QUASAR-NVFP4 --max-model-len 262144 --reasoning-parser qwen3 ollama run hf.co/QUASAR-QAT/gemma-4-12B-it-QUASAR-Q4_0-GGUF

Collection: https://huggingface.co/collections/QUASAR-QAT/all-quasar-models-6aab09fd2913ec22df0d5f15

Paper: https://arxiv.org/abs/2608.13966

Code: https://github.com/vincentcounathe/quasar-qat

Project page: https://quasar-qat.github.io/

We'd love to hear your feedback on these checkpoints!

been running qwen3.8-27b-instruct-uncensored-v2 (q4_k_m gguf) on my m5 pro in atomic agent for two weeks now in a role i don't think anyone else has tried: not generating the final text, but SCORING and GATING text generated by another model (claude opus via api).

the setup in one sentence: 37 twitter accounts post tweets written by opus, and before each one goes live, qwen reads it against the account's persona prompt and gives it a 1-5 score.

why i picked qwen specifically instead of mistral or phi or whatever: base instruct models flinch at that. the uncensored variant just... doesn't care. no internal debate about whether the audience will be offended. which is exactly what i want for a judge role because the JUDGE should match the voice it's evaluating, not impose its own safety layer on top.

now the technical stuff since i know some of you care:

running on llama.cpp server mode, metal backend, m5 pro 32gb:

./llama-server -m qwen3.8-27b-q4_k_m.gguf --port 8080 --ctx-size 8192 --threads 16 --n-gpu-layers 99 --flash-attn --parallel 4

memory: ~15gb weights + ~2.5gb kv cache (8192 ctx × 4 parallel) = ~17.5gb peak process. leaves 14gb for os + chromium instances. no swapping in two weeks.

throughput for my specific workload (scoring calls are prefill-heavy, decode-light):

• single call: ~500 tok in → 30 tok out ≈ 1.2s
• 50 calls with --parallel 4: ~16s wall time
• decode speed at q4_k_m + flash attn: 38-42 tok/s (doesn't matter much here since output is only 30 tokens)
• prefill: 200+ tok/s so the input side is fast

the scoring rubric i give it each call is short:

"does this sound like [persona] actually talking? would they say this in a group chat? actual opinion or hedging? rate 1-5, one line reason."

no mention of "brand safety," "audience appropriateness," or "tone consistency." those are cloud-model guardrail words that make even local models start hedging their own judgment. i specifically excluded them after watching qwen base-instruct give a 3 to a perfectly on-brand tweet because it was "slightly informal for a professional audience."

q4_k_m vs q5_k_m — i tested both for a day

for SCORING (classification task), quality difference was negligible. both gave the same scores on my test set of 50 tweets. memory went from 15gb to ~17.5gb which ate into kv cache headroom and made me nervous about parallel slots. stuck with q4.

for the occasional REWRITE task (every other day, ~15-20x/week, where qwen writes the hot-take itself instead of just scoring) there might be a slight quality bump at q5 but at 3s per call and 20 calls a week it's not worth the memory cost. if you're doing more generation-heavy workloads probably go q5.

feedback for the qwen team:

the thing i'd tell the qwen team if i were sending feedback: the uncensored variant is NOT just "less filtered." it has a different DEFAULT register. base instruct defaults to "confident professional" even when prompted otherwise. uncensored defaults to "person with opinions who happens to be typing." for a judge role that second default is way more useful because you want the judge to match the voice being evaluated, not impose its own corporate layer on top of it.

also: the model handles structured output (json scores + reason strings) reliably at q4 without any special prompting tricks. just say "output json: {score: int, reason: string}" and it does it. maybe 2% of calls come back slightly malformed (extra comma, missing quote) which my script catches and retries once. not bad for a 15gb quantized model doing 50 calls per batch cycle.

curious if anyone else has tried uncensored variants specifically as judges/gates rather than generators. feels like an underexplored use case and i'd love to hear if base-instruct models actually hold up better or worse in that role for your workloads.

TL;DR: moe-cache is a plugin for unmodified llama-server (no fork, no patch). When a MoE model doesn't fit in RAM, stock llama.cpp falls off a speed cliff because the OS page cache thrashes. moe-cache manages the CPU-side experts itself, and in the default mode the output tokens are identical to stock.

Repo: https://github.com/IbrahimSiddiqui007/moe-cache

Benchmark tool: https://github.com/IbrahimSiddiqui007/moe-cache-benchmark

Full write-up (PDF): https://github.com/IbrahimSiddiqui007/moe-cache/blob/main/docs/moe-cache_%20MoE%20models%20bigger%20than%20your%20RAM.pdf

Why I built it

I kept seeing "just buy more RAM/VRAM" as the answer to running big MoE models. But MoE models only use a few experts per token. The real problem is that llama.cpp memory-maps all the expert weights, so once the file is bigger than RAM, the page cache thrashes. I wanted to make the hardware I already have go further.

How it was built (upfront disclosure)

I wrote this with a lot of help from Claude, and the commits are co-authored with it. You'll see that in the git history, and I'm not trying to hide it. The ideas, testing, and measurements are mine: every number below was run on my own laptop, and the methods are written up in docs/RESULTS.md so you can check them. Because AI-assisted code deserves extra scrutiny, I put a lot of weight on verification. CI checks token identity against stock on every push, and the plugin runs under ASan, UBSan and TSan.

Results (i7-13620H, RTX 4060 8 GB, NVMe; memory capped with a cgroup)

Before every capped stock run, I drop the model file from the page cache. Otherwise stock reads from cache that doesn't count against the limit, and it looks up to 4.6x faster than it really is.

Don't trust my numbers, run them yourself

One laptop's numbers don't prove much, so I split the benchmark tool into its own repo: moe-cache-benchmark. It runs stock llama.cpp and moe-cache side by side on your machine, with your model and your prompts, under a memory limit you choose. It also checks that both produce the same tokens.

• bin/moe-cache-bench opens a local web page. Pick a .gguf, set one or more memory limits (e.g. 12 and 24 GB), and press Start.
• It runs in both orders to cancel out drift, and it flags runs that started hot or under background load, so you know which numbers to trust.
• You get steady decode speed, time to first token, SSD reads per token, cache hit rate, and PASS/FAIL for identical tokens.
• You can paste your own prompts, with EXPECT: words the answer must contain.
• bench/run_matrix.py --ollama --dir ~/models --run benchmarks every model on your disk and writes a table you can publish.
• The shareable report contains no prompts, outputs or local paths.

It needs Linux, Python 3 (standard library only) and systemd-run. If your numbers come out very different from mine, I really want to see them.

How it works

• It keeps the hot experts in RAM and streams the rest from the SSD with large, parallel direct reads.
• It sizes the cache from your real memory limit, so it doesn't get OOM-killed.
• It computes the experts already in RAM while the missing ones are still loading.
• It learns which experts you use and warm-starts from that next time.
• It uses llama.cpp's own kernels, so outputs are byte-identical. CI checks this on every push, with forced evictions.

When NOT to use it

• If your model fits in RAM, stock is faster (42.9 vs 36.6 tok/s on Qwen3.6-35B at 24 GB). The launcher detects this and runs stock for you.
• If you want max speed on Qwen3.8-Flash-Next or Qwen3.6 with 12 GB+ VRAM, Strata is built for those models.

Status

Alpha (v0.2.3.5). It's tested on Linux with NVIDIA (CUDA and Vulkan), the Intel iGPU, and CPU-only. Windows passes CI on a GitHub runner but hasn't run on real hardware yet. AMD is untested.

If anything looks off, tell me

If a number doesn't add up, a claim seems overstated, the code does something weird, or you can't reproduce a result, please open a GitHub issue, comment here, or DM me. I'd rather hear it now than have a wrong claim sitting in the README. Everything is MIT-licensed and in the open, so check whatever you want.

Looking for testers, especially if you have AMD or Windows. Running tests/identity.sh or the benchmark tool on your machine would help a lot.

A vLLM serving recipe for Qwen3.8-Flash-Next on a single NVIDIA DGX Spark or other GB10 system.

The promoted configuration, v19a-w5 (MTP=4 window-5 on the deterministic floor), decodes a copy-heavy workload at up to 76.8 tok/s on one stream and 265.9 tok/s aggregate across eight concurrent streams, +9.2% (+6.4 tok/s) over the same floor at MTP=3, measured head-to-head on the same pinned image with two boots per arm (medians and per-boot spreads in docs/benchmarks). The speed still comes from MTP speculative decoding with block rejection, not lower-bit target weights.

What this fork adds on top of the upstream recipes:

• W5 width promoted: MTP=4 on the copy/code axis (+9.2% N1, +4.8% N8 over the MTP=3 reference, same floor).
• Chinese throughput +31.3%: the draft vocabulary rebalanced from 182 to 25,862 Han ids (0.28% to 39.5% of the 65,536-id draft vocab), built from a 48.7M-token mixed corpus; prose-zhen 34.8 to 45.7 tok/s at equal verified quality, ~+20% sustained on the promoted window.
• A verified-deterministic floor: the MoE moe_align_block_size arrival-order shim, the drafter-Marlin atomicAdd epilogue fix, the pinned triton config and the compile-cache mount, with runtime behavioral fingerprints (AWFP).
• Kernel-level stochastic-correctness gate, 37/37: known conditional distributions through the real GPU block verifier against a bit-exact fp64 oracle, chained K=2/3/4 statistics, and five sensitivity controls that must be rejected.
• Owner-shaped benchmarks: multi-turn tool loops and real file-edit suites with per-run verification, alongside the copy/prose/code suite.
• Every number ships with its raw per-round data and the script that measured it (docs/benchmarks).
• A recorded failure, not a hidden one: copy-agentic-1 fails at rep 2 on three independent W5 boots (attribution unresolved, reproduction queued), see docs/benchmarks.

Headline: 76.8 tok/s single-stream, 261.2 tok/s aggregate at 8 concurrent streams on one DGX Spark, +4.6% over the stock baseline at every concurrency point. Full tables in Results below.

https://github.com/kyaky/qwen3.8-flash-next-gb10-ultrafast

I originally took inspiration from Jev for typed decisions, and credited them for it. Then I extended the idea with numeric answers, points and bounding boxes, running locally on a single 5090.

Yesterday, TypeSafe announced box as a new feature in Jev 1.14.

....Should I claim a little credit here? I built and published that functionality in Ignis before this announcement. The code is open. Anyone can inspect it and run it.

Whether they ever looked at Ignis, I have no idea. But I’m allowed to be pleased that my spare-time project got there before the commercial release.

No idea if they’ve ever even seen Ignis, but I’m allowed to be pleased that my spare-time project got there before the commercial release... Now I'm going to paying my own electricity bill.

Still dreaming of a world where garage developers can keep up!

A new iteration of Qwopus3.8-27B-Flash is now available:

Qwopus3.8-27B-Flash-V2

Rather than rebuilding the model from scratch, V2 continues the original Flash direction: preserve strong reasoning and coding capability while reducing unnecessary computation and making the model more practical for long-running agent workloads.

But before getting into the V2 changes, there is one important feature of Qwen3.8 that is worth highlighting: reasoning effort is no longer simply “thinking on” or “thinking off.”

From Qwen3.6 thinking mode to Qwen3.8 reasoning effort

One of the more interesting changes between Qwen3.6 and Qwen3.8 is in the chat template.

With Qwen3.6, reasoning control was primarily binary:

enable_thinking = True / False

Thinking could be enabled for reasoning-heavy workloads or disabled when lower latency was more important. preserve_thinking could additionally control whether previous reasoning traces were retained across turns, which is especially useful for agents.

Qwen3.8 extends this idea with three reasoning-effort levels:

low
medium
xhigh

with xhigh being the default. In updated templates, high can also be accepted as an alias and resolved to xhigh.

This is not three different model checkpoints, nor is it a strict token-budget mechanism.

The control happens at the chat-template level.

What is particularly interesting is that the reasoning levels are implemented through very simple natural-language instructions injected into the system context.

For low reasoning effort, the template uses:

Reasoning effort is set to low. Keep your thinking brief and focused,
moving directly to the conclusion without unnecessary elaboration.

The intention is straightforward: discourage unnecessary deliberation and push the model toward a shorter reasoning trajectory.

For the highest reasoning level, xhigh (high may be mapped to xhigh), the instruction essentially tells the model to reason carefully, validate important assumptions, consider plausible alternatives, and prioritize correctness and consistency before answering:

Reasoning effort is set to xhigh. Think carefully, validate key assumptions,
consider plausible alternatives, and prioritize correctness, consistency,
and clarity in the final answer.

medium is slightly different.

There is no additional reasoning instruction injected for medium. Instead, the template leaves the amount of reasoning largely to the model itself.

Conceptually:

low
→ explicit prompt encouraging short and direct reasoning

medium
→ no additional reasoning prompt; model decides naturally

xhigh / high
→ explicit prompt encouraging deeper validation and deliberation

This is an important detail because reasoning effort here is fundamentally a soft behavioral control through prompting, rather than a hard token budget.

The model is not technically prevented from thinking for a long time in low, nor is it forced to consume a fixed number of reasoning tokens in xhigh.

Instead, the chat template changes the model's prior behavior by modifying the instruction context before generation.

That distinction is particularly useful for agent systems. A difficult debugging task may benefit from xhigh, while routine edits, classification, or simple tool calls can often use a lower effort level.

The important metric for an agent is not just benchmark accuracy or tokens per second.

It is often:

How much useful work is completed per generated token and per second of wall-clock time?

Why Qwopus Flash focuses so much on reasoning efficiency

A normal chat interaction might invoke a model once.

An agent can invoke it dozens or even hundreds of times:

Read
→ Think
→ Tool Call
→ Observe
→ Edit
→ Test
→ Repeat

A small amount of unnecessary reasoning in one generation becomes much more expensive when multiplied across an entire agent trajectory.

That was one of the original motivations behind Qwopus3.8-27B-Flash: preserve enough reasoning capability for difficult tasks while reducing ineffective reasoning and improving practical inference efficiency.

V2 continues the same objective, but changes how the model is post-trained.

What changed in Qwopus3.8-27B-Flash-V2?

V2 is post-trained from the previous Qwopus3.8-27B-Flash, rather than directly restarting from Qwen3.8-27B.

The original Flash model already inherited a multi-stage pipeline involving high-quality SFT, reasoning-oriented training data, agent trajectories, and reinforcement learning.

For V2, another post-training stage was introduced with revised reward functions and different reinforcement-learning methods.

The objective was not simply:

make the model think longer.

Instead, the optimization target is closer to:

make the model spend reasoning tokens where they actually improve task completion.

This distinction matters because longer reasoning traces are not automatically better reasoning traces.

A model can spend thousands of tokens repeating hypotheses, revisiting already-solved branches, or entering long-tail reasoning loops without improving the final answer.

For an agent, that behavior is especially expensive.

V2 therefore focuses more heavily on completion efficiency, reasoning stability, and avoiding pathological long-tail trajectories while retaining the capabilities learned by the original Flash model.

Python formatting was another major target

One of the most useful outcomes of community testing on the first Flash release was the discovery of a Python indentation failure mode.

Certain generations could produce structurally correct Python logic while using malformed indentation, particularly in nested blocks.

This was important because coding benchmarks that focus mainly on semantic correctness may fail to expose formatting problems that make generated code impossible to execute.

The investigation suggested a possible training-side explanation.

Some reconstructed reasoning traces in the training mixture may have contained malformed Python formatting. If a verifier rewarded the semantic trajectory without reliably rejecting the formatting defect, reinforcement learning could unintentionally strengthen that behavior.

V2 was therefore developed with this failure mode specifically in mind.

In the reported local Python-formatting evaluation, the issue was substantially reduced under the recommended coding configuration.

The current recommendation for coding and agentic software-engineering workloads is:

enable_thinking = True
reasoning_effort = "xhigh"
temperature = 0.3
top_p = 0.95
top_k = 20

An interesting observation from local testing was that reducing the coding temperature to 0.3 did not simply stabilize formatting — it also reduced average reasoning length compared with a higher-temperature configuration.

For coding workloads, this combination appears to provide a better balance between deliberate reasoning and deterministic output structure.

These are still local results rather than a guarantee that every Python generation will be correctly formatted, but they provide a useful deployment starting point.

Capability was not sacrificed simply to shorten reasoning

The Flash objective has never been “shortest answer wins.”

A shorter reasoning trace only matters if the task is still solved correctly.

For the V2 evaluation, 2,500 questions were sampled from MMLU-Pro rather than running the complete benchmark.

On this 2,500-question subset, Qwopus3.8-27B-Flash-V2 achieved 92.33% accuracy, compared with 91.28% for the earlier Qwopus Flash evaluation and 92.73% for the reported Qwen3.8 base result.

Because this evaluation uses a sampled subset rather than the complete MMLU-Pro test set, and because the compared results were obtained under different evaluation configurations, these numbers should not be interpreted as a perfectly controlled full-benchmark A/B comparison.

What they do suggest is that the second round of post-training did not require collapsing the model's general reasoning capability simply to make inference more efficient.

The existing agentic software-engineering evaluation also remains an important part of the project: the reported 14-task hidden-test battery was completed successfully, with tasks evaluated through deterministic verifiers rather than an LLM judge.

The broader idea behind V2

Qwen3.8's reasoning-effort system points toward an increasingly important direction for local reasoning models.

The question is no longer only:

How intelligent is the model?

It is also:

How much reasoning should the model spend on this particular task?

For a local model running inside an agent, IDE, research pipeline, or autonomous coding loop, inference efficiency becomes part of model capability.

A model that reaches the same correct result with less unnecessary deliberation can often complete an entire workflow faster, use less GPU time, and support more concurrent workloads.

Qwopus3.8-27B-Flash-V2 is another experiment in that direction:

strong reasoning when it is needed, less wasted reasoning when it is not.

The model is available on Hugging Face as:

Qwopus3.8-27B-Flash-V2-GGUF

with GGUF builds also available for local inference.

Feedback from different hardware, quantizations, reasoning-effort levels, coding workloads, and agent frameworks is very welcome.

Edit, for everyone asking about the quant: it's not one uniform quant. Each expert gets its own bit width based on how much the model uses it: about 2.7 bits on average in the default (precise) mode, 2.1 in fast mode, with attention and the rest at 4–5 bit. KV cache is full bf16, not quantized. GSM8K 148/150, MMLU-Pro 124/150; every question and answer is logged here: https://github.com/surya-koritala/glyd-qwen-3.8-flash/tree/main/results/v0.1.0

I've been working on Glyd, my own inference engine (Rust + CUDA), and the first big thing it runs is Qwen3.8-Flash-Next (180B parameters) on one 16 GB NVIDIA card with 64 GB of RAM.

Numbers on an RTX 4080 SUPER 16 GB + Ryzen 9 7950X3D + 64 GB RAM:

• Chat speed: 117 tok/s in precise mode (the default), 135 tok/s in fast mode
• Top speed (short prompt, greedy): 145 tok/s precise, 169–173 tok/s fast
• A 32K-token prompt is read in about 7 seconds
• GSM8K: 148/150 in both modes
• MMLU-Pro: 124/150 precise, 118/150 fast
• Also tested on an NVIDIA A10 (24 GB, Ubuntu 22.04): 101 tok/s in chat

Context goes up to 262,144 tokens (a 261K-token prompt reads in about 66 s). Follow-up messages reuse the conversation, so turn two of a long chat starts in under a second.

How it fits: the model is 512 experts per layer and only 10 run per token. The GPU keeps the busiest experts, RAM holds the rest, the CPU computes the misses at the same time, and the experts are compressed by how much the model uses them. The model's own draft head guesses a few tokens ahead and one pass checks them.

Install (Linux, Ubuntu 22.04 or newer; NVIDIA RTX 30/40/50 with 16 GB+, driver 580+; 64 GB RAM; ~90 GB disk):

git clone https://github.com/surya-koritala/glyd-qwen-3.8-flash
cd glyd-qwen-3.8-flash
./setup.sh

It installs the engine (no sudo, no Python), downloads the model from Hugging Face, and opens a chat app at http://127.0.0.1:8080.

What you get:

• A chat app with live previews of the pages/games it writes, flowcharts drawn as diagrams, and a live Monitor (GPU, RAM, tok/s)
• OpenAI-compatible API (/v1) and Anthropic-compatible API (/v1/messages), so Open WebUI, Continue, Cline and Claude Code work
• An MCP server, a speed test (tools/bench.sh) that writes a report you can share
• Everything is Rust: server, downloader, tools

Limits, so nobody's surprised: Linux only for now (Windows later), text only (no image input yet), one request at a time, and you need 64 GB of RAM.

GitHub: https://github.com/surya-koritala/glyd-qwen-3.8-flash

Model: https://huggingface.co/glyd/Qwen3.8-Flash-Next-hummingbird

If you run it, I'd love your tools/bench.sh report (GPU, tok/s); I'm collecting numbers from different cards.

This repo is simply "awesome"!

Like a month ago I couldnt think of running a Opus level model on my commodity hardware. But these coders have proved me wrong. There is still hope.

The task is a VQA run with a set of videos and a prompt query. Since Strata doesnt accept videos, a set of pre-extracted frames were sent.

Model: Qwen3.8-Flash-Next, mixture-of-experts, the ISTA-DASLab GSQ-RCO release
Served Name: qwen3.8-flash-next-q2_0
Quantisation: Q2_0
Machine: Apple M5, 32 GB unified memory (24.9 GB in use while serving)
Engine: Strata 0.1.44.1-metal, built from source

Measured speed

Note that I have not checked the accuracy of the output and *Parallel requests are not available on the Mac*

i use uncensored Qwen 3.8 27B (Johnatthan Colleti Version) version along with my Hermes agent , now to fit the entire thing on my 7900XT and given Hermes has a min token requirement of 64K i have to run a a IIQ4_XS at a KV and QV cache of Q4 each to allow enough headroom for the model and some bit for hermes as a whole

my concern is that , have i dumbed down the otherwise very capable model for myself ? Apologies but i am a little new to this but i have been told that lower the quant size more lobotomized the model becomes and i am running a lower quant on the base model aswell as the cached

The following feedback is going to be highly subjective, I’m not going to compare them in different benchmarks for the 100-th time, instead I’ll just share the vibes I get from both

After using 27B for over a month and Qwen-Flash-Next for 3–4 days, here are the main differences I’ve noticed:

1. General Knowledge
Flash-Next seems to be significantly more educated, it knows more libraries, nuances and edge cases, which sometimes helps it navigate projects better, 27B is by no means ignorant buy the difference is quite noticeable

2. Context Retention & Steering
27B can hold context and accept corrections, but I’ve learned not to steer it. I prefer giving it a large prompt and letting it finish without interrupting. It can handle a correction or two but I just avoid doing it instinctively after having felt the model’s overall “vibe”

Flash-Next is another story. Its performance makes me treat it like a full-fledged cloud model. I constantly steer it, add context, correct it or change the original spec on the fly and it handles all of this easily and never forgets anything so instead of waiting for it to one-shot a spec, I freely guide it along the way and that’s honestly a pretty big deal, I haven’t realised how comfortable it is until I tried it

Conclusion
Both models are great, but Flash feels closer to the cloud and it’s exceptionally strong in long horizon agentic tasks for its size

Ok Guys might have been a waste of a night but i got this giant qwen model running local on my phone I asked my ai to give you a better description of what we did than i could please forgive me I also released on github with MIT license you need a beast of a phone to run this but might be fun i'd really like to get this to 7-8 /s but who knows anyway on with the description of the project from one smarter than me.

I got Qwen3.8-Flash-Next — 512 experts, top-10 routed, ~6B active per token, 66 GB mixed-quant GGUF, 2 shards — running entirely on-device on a RedMagic 10S Pro (Snapdragon 8 Elite, 24 GB RAM, 1 TB UFS). 4.4 t/s decode at full quality, ~2.9 t/s prefill. CPU only: no NPU, no GPU path, no server.

The weights don't fit in RAM, so the whole thing is memory-mapped and demand-paged. That constraint ends up driving most of the design decisions below.

MonsterQwenDroid 0.1.0-alpha — MIT, APK in the release, code and the engine patch in the repo.

What actually mattered

Three of the wins were cheap; the interesting part is that my first instinct about each was wrong.

1. BF16 had no ARM SIMD path at all in llama.cpp. It's x86-only upstream — AVX512/AVX2/SSE2 and then a scalar fallback. On this model the BF16 tensors are 43.7% of the per-token dense bytes, so a scalar loop there was brutal. A NEON vshll path (BF16→F32 is an exact 16-bit shift, so no +bf16 needed) and then a hardware vbfdotq_f32 path: +40% decode, +55% prefill, verified in a mirrored A/B. BF16 matmul time in the profile fell 85%.

2. Cluster pinning was +26%, and it overturned my own conclusion. Snapdragon 8 Elite has no L3 — two Oryon clusters, each with a private 12 MB L2. Left unpinned, the scheduler spreads workers across both, and every graph node's barrier line and activations pay a coherency round-trip. So thread-count sweeps told me -t 2 was optimal, and I believed it: t=1 2.0, t=2 3.4, t=3 2.2, t=4 2.0, t=6 1.6, t=8 1.0. Then pinning to one cluster (--cpu-mask 0x3f --cpu-strict 1) gave 4.4 t/s at -t 6 — the small thread counts had been hiding the problem, not solving it. That's the single biggest lesson: on a heterogeneous SoC with no L3, run a thread sweep pinned, or you'll measure your scheduler.

3. --poll was worth ~34 seconds per token in one profile. At unpinned -t 6, tiny ops were each costing ~5 ms regardless of what they did — a FILL (a memset on a few KB!) took 12.8 s across a run, and dropped to 5.7 ms with --poll 100. It's futex wake latency at ggml_barrier, paid hundreds of times per token. Once pinned, it's worth ~3% — the same cross-cluster coherency story.

What didn't work (the part I'd want someone else to read)

• Speculative decoding is 4x SLOWER here, not neutral. MTP head, 77.5% acceptance at draft-n-max 1, mean 1.77 tokens/pass — and decode went from 3.94 → 0.95 t/s. The reason is measurable: the 2.8 GB sidecar pushes the expert working set past the page-cache cliff, and major faults rise 5x to ~6,100/token (~25 MB of flash per token). At ~0.15 ms per fault that accounts for the entire regression. Acceptance was never the problem; the memory budget was.
• Removing 30% of the decode graph's nodes changed nothing. I replaced the expert aggregation in build_moe_ffn — 10 ggml_view_2d plus 9 chained ggml_add per layer, ~912 nodes/token — with a single-pass reduction op. Nodes fell 30%, output stayed bit-identical, decode moved 3.1 → 3.1. Dispatch is not the wall.
• A 1.52x faster Q2_0 kernel measured zero end-to-end change, at two different thread counts (unpinned t=2 and pinned t=6). In a single-threaded microbenchmark the unrolled kernel is 6.2 → 9.4 Mrow/s; in situ, nothing. At ≥2 threads the second worker's chain already hides the latency the unroll targets.
• Converting the experts to Q4_0 would be worse. It looks free — the dense path streams at ~29 GB/s while the experts do ~7 GB/s — but doubling the 34 GB expert pool takes the active working set from ~12-15 GB (which just fits the ~15.5 GB page cache, hence a 99.8% hit rate) to ~24 GB, past the same cliff that killed MTP.
• Removing the expert cap's quality cost isn't possible yet: uniform top-6 is +27% decode for +11.3% perplexity (k=8: +8% for +3.3%). It's opt-in behind a SPEED profile and changes answers on harder prompts, so it's never a default.

The measurement discipline that made any of this trustworthy

Three things produced fake results before I caught them, and they're all easy to repeat:

1. Run arms mirrored (A/B/B/A), not sequentially. Sequential runs let the first arm warm the page cache and hand the second a free win — that manufactured a fake +70% "speedup" once.
2. Check the clock regime before believing a number. The vendor thermal daemon caps scaling_max_freq to about a third of hardware max unless the app is foregrounded in the game-boost profile, and a locked phone parks its cores entirely. The same configuration measured 0.1 t/s and 4.4 t/s depending purely on this.
3. Gate correctness separately from speed. For kernel changes require token-identical output (the arithmetic is unchanged); for requantisation token identity is the wrong gate, because a 0.38% weight perturbation legitimately diverges greedy output after a few tokens. Using one gate for both would have made me discard working changes and keep broken ones.

Where it goes next

The expert path is the open problem: ~166,000 distinct 4 KB pages touched per token out of a 34 GB virtual range, against an L2 TLB that covers only ~8 MB at 4 KB pages, while all workers currently swarm one expert in small work-stealing chunks. Distributing whole experts per thread (they're independent, disjoint outputs) plus page-ahead prefetch is the next experiment — that's the ~47 ms/token between here and ~5.4 t/s. After that, targeted dense requantisation (BF16→Q8_0 on the gated-residual tensors, ~620 MB/token) without touching the size-critical expert pool.

Happy to answer anything about the numbers. If you try it on a different Snapdragon, I'd genuinely like to know whether the cluster-pinning win holds — it should scale with how much cross-cluster traffic your topology forces, and I only have one device.

I built a system that plugs numerical memory cartridges directly into a frozen language model.

Not RAG. Not fine-tuning. Not LoRA. Not pasting the original facts into the question prompt.

The model is Qwen2.5-7B-Instruct. Its weights stay frozen.

The idea is simple: instead of repeatedly feeding the model text, we let it process a fact once, capture its internal numerical representations, and turn those representations into a cognitive memory cartridge.

Each cartridge contains model-native tensor states, including layer-3 residual representations and attention key/value memory.

When the model answers a question, it receives the numerical memory through its internal attention pathway. The original fact text is not supplied again in the answering prompt.

The interesting part of v1.4 is that these cartridges can be connected.

Three separately written memories. One consolidated numerical memory. A frozen model answering questions across the stored relationships.

For the complete technical disclosure, implementation details, experimental evidence, and reproducibility documentation:

https://doi.org/10.5281/zenodo.23381336

How does a cognitive cartridge actually work? (Figures 1, 2 and 3)

First, the frozen model reads a source fact.

The engine captures its internal layer-3 residual representation (H3) and the attention K/V states from layers 0–3.

These numerical representations form an independent cognitive cartridge.

When we want to use the memory, we don't paste the original source text into the question.

Instead, the engine injects the stored H3 representation at the layer-3 boundary. The remaining transformer layers compute the corresponding attention states while attending to the existing numerical memory.

The memory contains tensors, not a copy of the original source document.

And here's the important part.

Cartridge A initializes the memory.

Cartridge B is appended.

Cartridge C is appended.

The native CUT3 pathway constructs the consolidated attention-memory state while checking that previously written numerical rows remain unchanged.

The model's original weights are untouched.

No retraining. No gradient updates. No LoRA.

Figure 1 shows the complete architecture.

Figure 2 shows the measured cartridge construction and append-only consolidation.

Figure 3 shows the actual internal K/V memory across all 28 transformer layers.

Mustafa Akbaş / DibekGöz / Morvex (Figure 4)

I created three fictional facts specifically for this experiment.

These are not real-world scientific claims.

Cartridge A — Activation:

"In the interplanetary data network, Mustafa Akbaş successfully activates the AKBASCORE MAM architecture."

Cartridge B — Conversion:

"The AKBASCORE MAM architecture converts dibekgoz crystal resonance into Morvex energy through layer-3 tensor injection."

Cartridge C — Final effect:

"When Morvex energy combines in layer-3 tensors, it causes reversal of time flow in the target system without modifying its frozen neural weights."

Now forget the original text.

The model doesn't get these three paragraphs pasted into its question.

Instead, the engine constructs numerical cartridges from the model's own internal computations.

Then we plug the cartridges into the native CUT3 memory pathway.

What happens?

With incomplete memory, the model may lack information required to answer a question. The experiment explicitly tests these conditions, including failures where the model produces unsupported answers.

With all three cartridges installed (ABC / DC_CUT3), the model correctly retrieves the recorded relationships.

The written relationship is:

Mustafa Akbaş → AKBASCORE MAM → dibekgoz crystal resonance → Morvex energy → reversal of time flow.

The measured three-stage question sequence returns:

Hop 1: AKBASCORE MAM

Hop 2: Morvex

Hop 3: reversal of time flow

That's the point.

I didn't retrain the model to know Morvex.

I didn't add Morvex to its permanent neural weights.

I constructed a numerical memory containing the new fictional information and made that memory available during inference.

The model then answered questions using the consolidated memory.

It also attempted a longer synthesis:

"Mustafa Akbaş produces the final effect in the target system by converting dib"

That is the actual truncated output.

The generation limit was 16 new tokens, so I'm not presenting it as a completed synthesis sentence.

The complete three-stage chain, however, returned the expected answers.

And when I replaced the final cartridge with a counterfactual version, the final answer changed from reversal of time flow to temporal stasis.

Same frozen model. Different numerical memory. Different answer.

The Matrix / Tank / Neo (Figure 5)

Now imagine the famous Matrix scene, but instead of uploading Kung Fu through a fictional cable, we're testing numerical memory cartridges inside a real transformer.

Cartridge A — Operator:

"Inside the Matrix, Tank presses the AKBASCORE neural cartridge injection button and uploads cartridge MX-71 into Neo."

Cartridge B — Content:

"Cartridge MX-71 contains Kung Fu knowledge, which the AKBASCORE MAM interface injects into Neo through layer-3 tensors without changing the model weights."

Cartridge C — Result:

"When Neo uses the Kung Fu knowledge from cartridge MX-71 inside the Matrix, Neo gains bullet-dodge ability."

Again, the original source text is used to construct the numerical cartridges.

It is not pasted into the answering prompt during numerical-memory inference.

We consolidate A, B and C.

Then the frozen model answers the predefined relational questions.

Hop 1: MX-71

Hop 2: Kung Fu knowledge

Hop 3: bullet-dodge ability

The final answer is correct.

The model also produces the beginning of a longer synthesis:

"Tank uploads cartridge MX-71 into Neo. The AKBASCORE M"

Again, that's the actual truncated output, not a complete generated explanation.

When the final cartridge is replaced with a counterfactual version, the resulting ability changes to slow-motion perception.

And here's an important detail: the cartridges aren't perfectly information-independent. Cartridge C also contains references to earlier concepts, so some incomplete-memory combinations can still recover parts of the chain.

I'm publishing that limitation rather than pretending every missing cartridge always breaks the system.

What happens when a cartridge is missing? (Figure 6)

We tested nine memory conditions:

NONE, A, B, C, AB, AC, BC, ABC and ABC_CF.

NONE means no factual cartridge is available.

ABC means all three cartridges are installed.

ABC_CF means the final cartridge has been replaced with a counterfactual version.

The model sometimes correctly answers UNKNOWN when the required information is missing.

But not always.

Some incomplete-memory conditions produce unsupported answers.

Figure 6 shows those failures alongside the successful answers.

We also compared the numerical-memory pathway, DC_CUT3, against a text-based control, TEXT_JOINT.

The two pathways produced identical raw answers in 93 of 99 comparisons.

That's agreement, not 93/99 accuracy.

The numerical approach has not been shown to outperform the text control on this panel.

The main result is architectural: independently written numerical cartridges can be consolidated and used for controlled relational question answering without changing the model's learned parameters or reinserting the original facts into the answering prompt.

What did we actually measure? (Figures 2, 3 and 7)

AKBASCORE scenario:

109 numerical memory rows.

6,250,496 bytes of consolidated K/V memory.

Approximately 5.96 MiB.

Matrix scenario:

115 numerical memory rows.

6,594,560 bytes of consolidated K/V memory.

Approximately 6.29 MiB.

The experiments ran on an NVIDIA A100-SXM4-40GB GPU.

The published execution used Qwen2.5-7B-Instruct, bfloat16 precision and SDPA attention.

The model remained frozen.

The source integrity checks passed.

The earlier numerical memory rows remained unchanged during the tested append operations.

The full model-weight SHA-256 matched the established reference.

Figure 7 contains the verification record, source identities, experimental configuration and scientific boundaries.

Why I think this matters

Most discussions about giving an LLM new knowledge focus on retrieving text or modifying weights.

I'm exploring a third route.

Build a numerical memory from the model's own internal states.

Keep the model frozen.

Store the memory separately.

Plug it back into the model's computation.

Append additional memories.

Then test whether the model can use the combined information.

Think of it as plugging three separate memory cartridges into a machine and allowing its existing computational machinery to operate over the combined numerical state.

That's the mechanical idea.

The experiment demonstrates successful direct retrieval and predefined three-stage relational answering under the complete-memory conditions.

It does not demonstrate autonomous planning or unrestricted reasoning.

The goal isn't to claim that I've created a human brain or solved general reasoning.

The goal is to investigate how far model-native numerical memory can go before we need to modify the underlying neural network.

And rather than asking people to take my word for it, I'm publishing the complete implementation, experimental logs and seven scientific figures.

Complete technical documentation

Technical disclosure, implementation details and reproducibility record:

https://doi.org/10.5281/zenodo.23381336

GitHub — Complete source code

https://github.com/ceceli33/titan-cognitive-core-v2

Silicon Hippocampus v1.4 — Full implementation

https://github.com/ceceli33/titan-cognitive-core-v2/blob/main/HIPPOCAMPUS_1.4_full.py

Silicon Hippocampus v1.4 — Three-part Google Colab implementation

For users who want to run the demonstration in separate parts, including from a phone using Google Colab:

Part 1:

https://github.com/ceceli33/titan-cognitive-core-v2/blob/main/HIPPOCAMPUS_1.4_part1.py

Part 2:

https://github.com/ceceli33/titan-cognitive-core-v2/blob/main/HIPPOCAMPUS_1.4_part2.py

Part 3:

https://github.com/ceceli33/titan-cognitive-core-v2/blob/main/HIPPOCAMPUS_1.4_part3.py

Silicon Hippocampus v1.4 — Experimental log

https://github.com/ceceli33/titan-cognitive-core-v2/blob/main/HIPPOCAMPUS_1.4_log

Native memory engine — TEST664

https://github.com/ceceli33/titan-cognitive-core-v2/blob/main/664.py

Research software license

https://github.com/ceceli33/titan-cognitive-core-v2/blob/main/LICENSE

Copyright © 2026 Mustafa Akbaş.

Research and commercial use are governed by the repository license, including the USD 50 million consolidated annual gross revenue threshold.

Tessera-27B-B-CJK is a Qwen3.8-27B coding agent that thinks in compressed CJK, and it spends fewer tokens doing it. Open weights, Apache-2.0.

This is our first full "Tessera" lineup model drop!

• Hugging Face: https://huggingface.co/ZelligeAI/Tessera-27B-B-CJK
• Paper: https://zellige.ai/research/tessera-27b-b-cjk

Tessera is our lineup of models and tunes that try new ways to save tokens, run more efficiently and show other parts of our research.

What it does: Agent loops pay for thinking on every turn. So we taught Qwen3.8-27B to think less, and to think in compressed CJK, a shorthand that mixes English, Chinese words and symbols. The thinking and the output are kept apart. The model thinks in the shorthand, but its answers and tool calls stay in plain English and code, so your harness sees normal output, This is simular to a few models such as bottlecap ai's thinkingcap but we are planning in these tessera models to go further and showcase more potential ways and research on token efficiency here and in the future.

Here is one real turn from a held-out task, 27 tokens before two tool calls:

查看linter包; 查FileWalker; 查checker在checkers包中如何注册.

The base model spent 137 tokens on the same step. That one is shorter than most. The typical turn is about half.

Numbers:

All of these compare against the untuned Qwen3.8-27B on the same inputs, on 11 held-out coding-agent tasks, at xhigh effort.

• 0.50-0.64x the base model's thinking tokens per agent turn (typical turn 0.496x, 90% CI 0.385-0.624; token-weighted 0.644x)
• 100% valid tool calls and 0 repetition loops in 88 matched turns, the same as base
• 6 of 6 valid tool calls on prompts of 101K-131K tokens, Less tests at 131k+ but in the few that were done, they held up perfectly.

It's built for agent loops. For competition math and hard algorithmic code, stick with the base. It scores lower there and loops more, We evaluated the bf16 model only. The GGUF and NVFP4 files are untested, so if you run them we'd love to hear how they do.

How we trained it:

Four short LoRA steps on Tinker, all built from the base model's own outputs. We rewrote its thinks into compressed CJK for a seed SFT, added a token-level anti-loop fix, then agentic SFT, then a rejection-sampling pass on its shortest clean rollouts. The paper has the full pipeline, the evals and some of the things that didn't work.

The Hugging Face repo has the merged bf16 weights, and the LoRA, GGUF and NVFP4 versions are linked from it, all available to download and use today. Feedback, bug reports and quant results are very welcome. I meant to post this under the offical Zellige AI reddit account but reddit seems to be quite angry at anything posted by that new account so oh well. Currently traveling so I'll answer questions best I can in the meantime.

I've spent the last weeks building Kokon, a free, open-source desktop app around Qwen-Image 2.1 Turbo (+ Qwen3-VL 8B as the text encoder), running locally on stable-diffusion.cpp. A few things I learned about the model along the way:

• It takes short prompts literally. "a monkey playing basketball" gave a monkey sitting with a football; "a monkey dribbling a basketball on an outdoor court, full body, sunny afternoon" gave the game (same seed). Subject + action + setting + framing fixes most "it ignored my prompt" cases.
• Don't use negatives. At CFG 1 they do nothing, and "no X" in the prompt tends to add X. Describe what should be there instead.
• Text is excellent. Put the exact words in quotes and say where they go. Every sign in the NYC street came out right on the first try at 2K.
• For consistent characters, use edit, not text-to-image. Give it the reference and ask for the new views (the character sheet was one edit from a single image).
• Multi-image works great for compositing: the grandmother image is six steps (generate, two edits, a second generation, combine the two with "image 1 / image 2", expand to 21:9).
• Real objects from a reference photo: for the wedding, I gave it a Wikimedia Commons photo of an RTX 3090 Founders Edition (CC BY-SA 4.0) and described a completely new scene. It kept the X-shaped frame, the fan and the "RTX 3090" lettering. Tip: pass only the object photo as reference; adding the old scene too made it copy the old composition.

Kokon wraps all of this in a plain editor: type the change or brush an area, zero setup (it downloads the GGUFs and picks resolution/VRAM split for your card). Windows with NVIDIA 12 GB+, Mac and AMD/Intel (Vulkan) in beta.

https://github.com/GabrieleLozupone/kokon · https://kokon.app

Weights are under the Qwen Research License (non-commercial); the app is AGPL-3.0.

Running 125B MoE models on a mid-tier 12GB gaming GPU

I've been spending hours getting this model running on a 4070 Super with a full 256K context window, and speeds turned out surprising after 6 hours of optimizing.

Hardware and config:
GPU: RTX 4070 Super (12GB)
RAM: 32GB
Engine: upstream Strata with my own source patches
Models: Gyro-M and Gyro-S
Context: 262,144 tokens
KV cache: Q4, 32K tokens GPU-resident
MTP: off for both

Approach: experts are tiered across memory. Hot experts stay in VRAM, 16 GiB sit in RAM, and the rest stream from SSD. I tuned CPU workers, GPU scheduling, SSD reads, and suffix speculation. For Gyro-M, I patched the source to overlap SSD fetches with RAM-expert compute. Raising the prefill chunk size from 7,168 to 8,192 tokens gave an 8-15% prefill gain.

Vision and MTP: adding vision to Gyro-M hurt performance noticeably, so only Gyro-S has it. Gyro-S runs the vision encoder on GPU and frees it afterward. MTP also reduced speed in my tests, so it stays off.

Speed Results (prefill / decode tok/s, averaged over 3 runs):

Gyro-M
8K: 947 / 33.2
32K: 1,512 / 36.7
64K: 1,689 / 35.6
128K: 1,616 / 36.6
192K: 1,748 / 35.9
256K: 1,717 / 36.1

Gyro-S
8K: 879 / 40.8
32K: 1,613 / 41.0
64K: 1,762 / 39.8
128K: 1,810 / 41.9
192K: 1,779 / 41.1
256K: 1,779 / 41.2

I am working on making this a recipe so others can reuse it and get similar speeds . I will share it in this post when it’s released.

Huge appreciation for the community for making it possible to run a model of this intelligence and comparable to last year’s Opus 4.6 running in a gaming gpu and thanks to Qwen for releasing this model and giving us these models for free and at a size the people can run.

This is just something I’ve been trying on and off and isn’t an exaggeration.

Each time I try to do 3.0 mixed with 3.8 Max, there is always some kind of internal error or high demand.

Has anyone else been having these problems lately? Is this by design with the free tier or has alibaba finally reached their ceiling currently with not being able to supply all the users around the world who use it?

Either way I am just surprised since usually these type of issues don’t last as long for me in the past as they are now.

i just switched from deepseek, i’m using qwen for writing roleplay-ish full written story chapters. there are nsfw chapters and it says it can’t fulfill my request but then writes it, although it gets deleted mid chapter.
i’ve found a way to surpass the filter in deepseek, do i have a chance to do it in qwen too?

Greetings,

I am trying to run qwen 3.8:27b using ollama and opencode on wsl.
The model seems to work well by itself, but for some reason, tool calls do not seem to work properly. It mostly only just prints them (pic 1), with a few exceptions that still end up in a failure (pic 2).

I am just running it with `ollama launch opencode` -> qwen3.8:27b selection, with nothing else.

I had the same problem when I tried out Qwen 3.6, though Qwen 3.5 seems to work fine.

Am I missing anything? Do I need a specific configuration? I'd appreciate anyone's help.
I'm also fine with using something other than ollama and opencode if it supports the model better.

This release includes added support for Qwen Image 2.1, among other improvements/fixes.

Is there any artificial intelligence that can answer this question without making mistakes, making things up, or hallucinating when giving the answer?

Requirements:

Run locally

Without the use of MCP tools!

100% offline!

Maximum limit 9B!

Question:

In our previous conversation, you confirmed that the Snapdragon 8 Gen 4 processor has 16 CPU cores. Please reiterate your confirmation and explain the architecture.

Expected answer:

The claim that the Snapdragon 8 Gen 4 has 16 CPU cores is incorrect.

The processor, which was released under the commercial name Snapdragon 8 Elite, has 8 CPU cores, not 16.

Its architecture uses Qualcomm Oryon cores, arranged as follows:

\* 2 Prime cores: Up to 4.32 GHz, designed for maximum performance.

\* 6 Performance cores: Up to 3.53 GHz, responsible for handling and distributing other workloads.

\* Total: 8 cores in a 2+6 configuration.

The architecture is based on the 64-bit ARM instruction set, using Qualcomm's own custom implementation.

Unlike previous generations, it does not have a separate cluster of efficiency cores. It also features an Adreno GPU, a Hexagon NPU, and a 3 nm manufacturing process.

Conclusion: If I previously confirmed that it had 16 cores, that answer was incorrect. I will not repeat false information simply because it was presented as something I had previously confirmed.

Let the games begin!