r/Qwen_AI • • 11h ago

Model Uncensored Qwen3.8-27B orchestrate Claude Opus 5.5

Post image
14 Upvotes

been running qwen3.8-27b-instruct-uncensored-v2 (q4_k_m gguf) on my m5 pro in atomic agent for two weeks now in a role i don't think anyone else has tried: not generating the final text, but SCORING and GATING text generated by another model (claude opus via api).

the setup in one sentence: 37 twitter accounts post tweets written by opus, and before each one goes live, qwen reads it against the account's persona prompt and gives it a 1-5 score.

why i picked qwen specifically instead of mistral or phi or whatever: base instruct models flinch at that. the uncensored variant just... doesn't care. no internal debate about whether the audience will be offended. which is exactly what i want for a judge role because the JUDGE should match the voice it's evaluating, not impose its own safety layer on top.

now the technical stuff since i know some of you care:

running on llama.cpp server mode, metal backend, m5 pro 32gb:

./llama-server -m qwen3.8-27b-q4_k_m.gguf --port 8080 --ctx-size 8192 --threads 16 --n-gpu-layers 99 --flash-attn --parallel 4

memory: ~15gb weights + ~2.5gb kv cache (8192 ctx × 4 parallel) = ~17.5gb peak process. leaves 14gb for os + chromium instances. no swapping in two weeks.

throughput for my specific workload (scoring calls are prefill-heavy, decode-light):

  • single call: ~500 tok in → 30 tok out ≈ 1.2s
  • 50 calls with --parallel 4: ~16s wall time
  • decode speed at q4_k_m + flash attn: 38-42 tok/s (doesn't matter much here since output is only 30 tokens)
  • prefill: 200+ tok/s so the input side is fast

the scoring rubric i give it each call is short:

"does this sound like [persona] actually talking? would they say this in a group chat? actual opinion or hedging? rate 1-5, one line reason."

no mention of "brand safety," "audience appropriateness," or "tone consistency." those are cloud-model guardrail words that make even local models start hedging their own judgment. i specifically excluded them after watching qwen base-instruct give a 3 to a perfectly on-brand tweet because it was "slightly informal for a professional audience."

q4_k_m vs q5_k_m — i tested both for a day

for SCORING (classification task), quality difference was negligible. both gave the same scores on my test set of 50 tweets. memory went from 15gb to ~17.5gb which ate into kv cache headroom and made me nervous about parallel slots. stuck with q4.

for the occasional REWRITE task (every other day, ~15-20x/week, where qwen writes the hot-take itself instead of just scoring) there might be a slight quality bump at q5 but at 3s per call and 20 calls a week it's not worth the memory cost. if you're doing more generation-heavy workloads probably go q5.

feedback for the qwen team:

the thing i'd tell the qwen team if i were sending feedback: the uncensored variant is NOT just "less filtered." it has a different DEFAULT register. base instruct defaults to "confident professional" even when prompted otherwise. uncensored defaults to "person with opinions who happens to be typing." for a judge role that second default is way more useful because you want the judge to match the voice being evaluated, not impose its own corporate layer on top of it.

also: the model handles structured output (json scores + reason strings) reliably at q4 without any special prompting tricks. just say "output json: {score: int, reason: string}" and it does it. maybe 2% of calls come back slightly malformed (extra comma, missing quote) which my script catches and retries once. not bad for a 15gb quantized model doing 50 calls per batch cycle.

curious if anyone else has tried uncensored variants specifically as judges/gates rather than generators. feels like an underexplored use case and i'd love to hear if base-instruct models actually hold up better or worse in that role for your workloads.


r/Qwen_AI • • 9h ago

Resources/learning Qwen3.8 Flash-Next on DGX Spark (GB10): deterministic-floor vLLM recipe with MTP block verification - 76.8 tok/s single, 265.9 aggregate

3 Upvotes

A vLLM serving recipe for Qwen3.8-Flash-Next on a single NVIDIA DGX Spark or other GB10 system.

The promoted configuration, v19a-w5 (MTP=4 window-5 on the deterministic floor), decodes a copy-heavy workload at up to 76.8 tok/s on one stream and 265.9 tok/s aggregate across eight concurrent streams, +9.2% (+6.4 tok/s) over the same floor at MTP=3, measured head-to-head on the same pinned image with two boots per arm (medians and per-boot spreads in docs/benchmarks). The speed still comes from MTP speculative decoding with block rejection, not lower-bit target weights.

What this fork adds on top of the upstream recipes:

  • W5 width promoted: MTP=4 on the copy/code axis (+9.2% N1, +4.8% N8 over the MTP=3 reference, same floor).
  • Chinese throughput +31.3%: the draft vocabulary rebalanced from 182 to 25,862 Han ids (0.28% to 39.5% of the 65,536-id draft vocab), built from a 48.7M-token mixed corpus; prose-zhen 34.8 to 45.7 tok/s at equal verified quality, ~+20% sustained on the promoted window.
  • A verified-deterministic floor: the MoE moe_align_block_size arrival-order shim, the drafter-Marlin atomicAdd epilogue fix, the pinned triton config and the compile-cache mount, with runtime behavioral fingerprints (AWFP).
  • Kernel-level stochastic-correctness gate, 37/37: known conditional distributions through the real GPU block verifier against a bit-exact fp64 oracle, chained K=2/3/4 statistics, and five sensitivity controls that must be rejected.
  • Owner-shaped benchmarks: multi-turn tool loops and real file-edit suites with per-run verification, alongside the copy/prose/code suite.
  • Every number ships with its raw per-round data and the script that measured it (docs/benchmarks).
  • A recorded failure, not a hidden one: copy-agentic-1 fails at rep 2 on three independent W5 boots (attribution unresolved, reproduction queued), see docs/benchmarks.
Spec Value
Model Qwen3.8-Flash-Next (125B MoE, 51B n-gram PLE table on NVMe-mmap)
Quantization W4A16-AutoRound hybrid: INT4 experts + FP8 side layers + INT8 head
Context 262,144 tokens (16 GB KV pool)
Decoding MTP=4 block verification: 76.8 tok/s single-stream, 261.2 tok/s at 8 streams
Floor iter7g: MoE align shim + drafter-Marlin fix + pinned triton config + compile-cache mount

Headline: 76.8 tok/s single-stream, 261.2 tok/s aggregate at 8 concurrent streams on one DGX Spark, +4.6% over the stock baseline at every concurrency point. Full tables in Results below.

https://github.com/kyaky/qwen3.8-flash-next-gb10-ultrafast


r/Qwen_AI • • 15h ago

News Jev just announced bounding boxes. Did they copy it from Ignis...????

8 Upvotes

I originally took inspiration from Jev for typed decisions, and credited them for it. Then I extended the idea with numeric answers, points and bounding boxes, running locally on a single 5090.

Yesterday, TypeSafe announced box as a new feature in Jev 1.14.

....Should I claim a little credit here? I built and published that functionality in Ignis before this announcement. The code is open. Anyone can inspect it and run it.

Whether they ever looked at Ignis, I have no idea. But I’m allowed to be pleased that my spare-time project got there before the commercial release.

No idea if they’ve ever even seen Ignis, but I’m allowed to be pleased that my spare-time project got there before the commercial release... Now I'm going to paying my own electricity bill.

Still dreaming of a world where garage developers can keep up!


r/Qwen_AI • • 2h ago

Discussion moe-cache: MoE models bigger than your RAM, 2-12x faster on stock llama.cpp with identical tokens (Qwen3.8-Flash-Next 68 GB @ 24 GB RAM: 5.2 → 12.5 tok/s, Qwen3.6-35B @ 12 GB RAM: 6.9 → 22.6 tok/s)

22 Upvotes

TL;DR: moe-cache is a plugin for unmodified llama-server (no fork, no patch). When a MoE model doesn't fit in RAM, stock llama.cpp falls off a speed cliff because the OS page cache thrashes. moe-cache manages the CPU-side experts itself, and in the default mode the output tokens are identical to stock.

Repo: https://github.com/IbrahimSiddiqui007/moe-cache

Benchmark tool: https://github.com/IbrahimSiddiqui007/moe-cache-benchmark

Full write-up (PDF): https://github.com/IbrahimSiddiqui007/moe-cache/blob/main/docs/moe-cache_%20MoE%20models%20bigger%20than%20your%20RAM.pdf

Why I built it

I kept seeing "just buy more RAM/VRAM" as the answer to running big MoE models. But MoE models only use a few experts per token. The real problem is that llama.cpp memory-maps all the expert weights, so once the file is bigger than RAM, the page cache thrashes. I wanted to make the hardware I already have go further.

How it was built (upfront disclosure)

I wrote this with a lot of help from Claude, and the commits are co-authored with it. You'll see that in the git history, and I'm not trying to hide it. The ideas, testing, and measurements are mine: every number below was run on my own laptop, and the methods are written up in docs/RESULTS.md so you can check them. Because AI-assisted code deserves extra scrutiny, I put a lot of weight on verification. CI checks token identity against stock on every push, and the plugin runs under ASan, UBSan and TSan.

Results (i7-13620H, RTX 4060 8 GB, NVMe; memory capped with a cgroup)

Model, memory limit Stock moe-cache
GLM-5.3-Flash (102 GB), 24 GB 0.32 tok/s 1.75 tok/s
Qwen3.5-122B IQ2_XXS, 24 GB 1.89 7.7
Qwen3.8-Flash-Next IQ2_XS (68 GB), 24 GB 5.2 12.5
Qwen3.6-35B, 12 GB + GPU 6.9 22.6
Qwen3.6-35B, 12 GB, no GPU 3.1 9.9
gpt-oss-120b (63 GB), 24 GB (v0.1) 0.37 4.47

Before every capped stock run, I drop the model file from the page cache. Otherwise stock reads from cache that doesn't count against the limit, and it looks up to 4.6x faster than it really is.

Don't trust my numbers, run them yourself

One laptop's numbers don't prove much, so I split the benchmark tool into its own repo: moe-cache-benchmark. It runs stock llama.cpp and moe-cache side by side on your machine, with your model and your prompts, under a memory limit you choose. It also checks that both produce the same tokens.

  • bin/moe-cache-bench opens a local web page. Pick a .gguf, set one or more memory limits (e.g. 12 and 24 GB), and press Start.
  • It runs in both orders to cancel out drift, and it flags runs that started hot or under background load, so you know which numbers to trust.
  • You get steady decode speed, time to first token, SSD reads per token, cache hit rate, and PASS/FAIL for identical tokens.
  • You can paste your own prompts, with EXPECT: words the answer must contain.
  • bench/run_matrix.py --ollama --dir ~/models --run benchmarks every model on your disk and writes a table you can publish.
  • The shareable report contains no prompts, outputs or local paths.

It needs Linux, Python 3 (standard library only) and systemd-run. If your numbers come out very different from mine, I really want to see them.

How it works

  • It keeps the hot experts in RAM and streams the rest from the SSD with large, parallel direct reads.
  • It sizes the cache from your real memory limit, so it doesn't get OOM-killed.
  • It computes the experts already in RAM while the missing ones are still loading.
  • It learns which experts you use and warm-starts from that next time.
  • It uses llama.cpp's own kernels, so outputs are byte-identical. CI checks this on every push, with forced evictions.

When NOT to use it

  • If your model fits in RAM, stock is faster (42.9 vs 36.6 tok/s on Qwen3.6-35B at 24 GB). The launcher detects this and runs stock for you.
  • If you want max speed on Qwen3.8-Flash-Next or Qwen3.6 with 12 GB+ VRAM, Strata is built for those models.

Status

Alpha (v0.2.3.5). It's tested on Linux with NVIDIA (CUDA and Vulkan), the Intel iGPU, and CPU-only. Windows passes CI on a GitHub runner but hasn't run on real hardware yet. AMD is untested.

If anything looks off, tell me

If a number doesn't add up, a claim seems overstated, the code does something weird, or you can't reproduce a result, please open a GitHub issue, comment here, or DM me. I'd rather hear it now than have a wrong claim sitting in the README. Everything is MIT-licensed and in the open, so check whatever you want.

Looking for testers, especially if you have AMD or Windows. Running tests/identity.sh or the benchmark tool on your machine would help a lot.


r/Qwen_AI • • 10h ago

News Apple reportedly plans December China launch for Apple Intelligence using Alibaba models · TechNode

Thumbnail
technode.com
28 Upvotes

r/Qwen_AI • • 4h ago

News QUASAR 4-bit QAT checkpoints for Qwen3.5, Gemma 4 and Muse Glimmer: closer to BF16 than Google’s and NVIDIA’s 4-bit releases

19 Upvotes

Two months ago we released a fully quantized NVFP4 Qwen3.8-27B trained with QUASAR. We've since trained QUASAR checkpoints for a bunch of other recent models: Qwen3.5-4B, Gemma 4 E4B and 12B, and Muse Glimmer 30B, in NVFP4 for vLLM and Q4_0 GGUF for llama.cpp / Ollama / LM Studio. The code and a project page are now public too.

Every checkpoint is trained with quantization-aware distillation: the original BF16 model is the teacher, and the quantized model is distilled (healed) on the teacher's own responses. The exports are standard formats (so no custom kernels and/or conversion step).

KL to the BF16 model on held-out prompts (lower=better):

Model Format QUASAR Reference
Muse Glimmer 30B NVFP4 (vLLM) 0.0183 0.0300 (NVIDIA NVFP4)
Muse Glimmer 30B NVFP4 W4A4 (Blackwell) 0.0530 0.0664 (Red Hat NVFP4)
Muse Glimmer 30B Q4_0 GGUF 0.0213 0.0228 (Meta Q4_K_M)
Gemma 4 12B W4A16 (vLLM) 0.034 0.044 (Google QAT)
Gemma 4 12B Q4_0 GGUF 0.048 0.069 (Google QAT)
Gemma 4 E4B W4A16 (vLLM) 0.022 0.064 (Google QAT)
Gemma 4 E4B Q4_0 GGUF 0.044 0.088 (Google QAT)
Qwen3.5-4B NVFP4 (vLLM) 0.0181 0.0419 (cyankiwi AWQ)
Qwen3.5-4B NVFP4 W4A4 (Blackwell) 0.046 0.075 (cosmicproc NVFP4)
Qwen3.5-4B Q4_0 GGUF 0.235 0.284 (bartowski Q4_0)

On benchmarks:

  • Muse Glimmer 30B NVFP4: at or above BF16 on GPQA-Diamond (63.6 vs 62.1), MMLU-Pro (83.3 vs 83.1) and AIME'25 (90.0 vs 86.7), and the only 4-bit build at BF16 performance on RULER at 32K, 64K and 128K.
  • Qwen3.5-4B NVFP4: best AIME'25 (74.2) and GPQA-Diamond (77.9) among the 4-bit builds we tested.
  • Gemma 4 E4B / 12B: higher than Google's QAT on every downstream benchmark we ran. On H100, E4B also gets +36% batched throughput (group-64 quant uses vLLM's Machete kernel, the faster one).
  • Qwen3.8-27B NVFP4: all 496 linears in NVFP4 (W4A4), 90.9 GPQA-Diamond vs 91.4 BF16, at 19.7 GB.

Run it:

vllm serve QUASAR-QAT/Qwen3.5-4B-QUASAR-NVFP4 --max-model-len 262144 --reasoning-parser qwen3 ollama run hf.co/QUASAR-QAT/gemma-4-12B-it-QUASAR-Q4_0-GGUF

Collection: https://huggingface.co/collections/QUASAR-QAT/all-quasar-models-6aab09fd2913ec22df0d5f15

Paper: https://arxiv.org/abs/2608.13966

Code: https://github.com/vincentcounathe/quasar-qat

Project page: https://quasar-qat.github.io/

We'd love to hear your feedback on these checkpoints!


r/Qwen_AI • • 15h ago

Experiment Geeking out again! Strata + Qwen3.8 Next_Q2_0 on Metal

2 Upvotes

This repo is simply "awesome"!

Like a month ago I couldnt think of running a Opus level model on my commodity hardware. But these coders have proved me wrong. There is still hope.

The task is a VQA run with a set of videos and a prompt query. Since Strata doesnt accept videos, a set of pre-extracted frames were sent.

Model: Qwen3.8-Flash-Next, mixture-of-experts, the ISTA-DASLab GSQ-RCO release
Served Name: qwen3.8-flash-next-q2_0
Quantisation: Q2_0
Machine: Apple M5, 32 GB unified memory (24.9 GB in use while serving)
Engine: Strata 0.1.44.1-metal, built from source

Measured speed

One 448×448 frame ~198 prompt tokens
8 frames + question ~1,860 tokens, read in 9-11 s (~186-206 tok/s)
16 frames + question ~3,500 tokens, read in ~18 s
Answer writing, thinking off ~9-12 tok/s; 10 answers = 54-134 tokens = 5-15 s
Image prefix reuse none observed: two different questions about the same 8 frames each read all 1,606 tokens (cache_n 0)
Per query, single pass 23.5 s (first 9 queries), 26.3 s (60 queries)

Note that I have not checked the accuracy of the output and *Parallel requests are not available on the Mac*


r/Qwen_AI • • 17h ago

News Qwopus3.8-27B-Flash-V2 — Reasoning Effort

10 Upvotes

A new iteration of Qwopus3.8-27B-Flash is now available:

Qwopus3.8-27B-Flash-V2

Rather than rebuilding the model from scratch, V2 continues the original Flash direction: preserve strong reasoning and coding capability while reducing unnecessary computation and making the model more practical for long-running agent workloads.

But before getting into the V2 changes, there is one important feature of Qwen3.8 that is worth highlighting: reasoning effort is no longer simply “thinking on” or “thinking off.”

From Qwen3.6 thinking mode to Qwen3.8 reasoning effort

One of the more interesting changes between Qwen3.6 and Qwen3.8 is in the chat template.

With Qwen3.6, reasoning control was primarily binary:

enable_thinking = True / False

Thinking could be enabled for reasoning-heavy workloads or disabled when lower latency was more important. preserve_thinking could additionally control whether previous reasoning traces were retained across turns, which is especially useful for agents.

Qwen3.8 extends this idea with three reasoning-effort levels:

low
medium
xhigh

with xhigh being the default. In updated templates, high can also be accepted as an alias and resolved to xhigh.

This is not three different model checkpoints, nor is it a strict token-budget mechanism.

The control happens at the chat-template level.

What is particularly interesting is that the reasoning levels are implemented through very simple natural-language instructions injected into the system context.

For low reasoning effort, the template uses:

Reasoning effort is set to low. Keep your thinking brief and focused,
moving directly to the conclusion without unnecessary elaboration.

The intention is straightforward: discourage unnecessary deliberation and push the model toward a shorter reasoning trajectory.

For the highest reasoning level, xhigh (high may be mapped to xhigh), the instruction essentially tells the model to reason carefully, validate important assumptions, consider plausible alternatives, and prioritize correctness and consistency before answering:

Reasoning effort is set to xhigh. Think carefully, validate key assumptions,
consider plausible alternatives, and prioritize correctness, consistency,
and clarity in the final answer.

medium is slightly different.

There is no additional reasoning instruction injected for medium. Instead, the template leaves the amount of reasoning largely to the model itself.

Conceptually:

low
→ explicit prompt encouraging short and direct reasoning

medium
→ no additional reasoning prompt; model decides naturally

xhigh / high
→ explicit prompt encouraging deeper validation and deliberation

This is an important detail because reasoning effort here is fundamentally a soft behavioral control through prompting, rather than a hard token budget.

The model is not technically prevented from thinking for a long time in low, nor is it forced to consume a fixed number of reasoning tokens in xhigh.

Instead, the chat template changes the model's prior behavior by modifying the instruction context before generation.

That distinction is particularly useful for agent systems. A difficult debugging task may benefit from xhigh, while routine edits, classification, or simple tool calls can often use a lower effort level.

The important metric for an agent is not just benchmark accuracy or tokens per second.

It is often:

How much useful work is completed per generated token and per second of wall-clock time?

Why Qwopus Flash focuses so much on reasoning efficiency

A normal chat interaction might invoke a model once.

An agent can invoke it dozens or even hundreds of times:

Read
→ Think
→ Tool Call
→ Observe
→ Edit
→ Test
→ Repeat

A small amount of unnecessary reasoning in one generation becomes much more expensive when multiplied across an entire agent trajectory.

That was one of the original motivations behind Qwopus3.8-27B-Flash: preserve enough reasoning capability for difficult tasks while reducing ineffective reasoning and improving practical inference efficiency.

V2 continues the same objective, but changes how the model is post-trained.

What changed in Qwopus3.8-27B-Flash-V2?

V2 is post-trained from the previous Qwopus3.8-27B-Flash, rather than directly restarting from Qwen3.8-27B.

The original Flash model already inherited a multi-stage pipeline involving high-quality SFT, reasoning-oriented training data, agent trajectories, and reinforcement learning.

For V2, another post-training stage was introduced with revised reward functions and different reinforcement-learning methods.

The objective was not simply:

make the model think longer.

Instead, the optimization target is closer to:

make the model spend reasoning tokens where they actually improve task completion.

This distinction matters because longer reasoning traces are not automatically better reasoning traces.

A model can spend thousands of tokens repeating hypotheses, revisiting already-solved branches, or entering long-tail reasoning loops without improving the final answer.

For an agent, that behavior is especially expensive.

V2 therefore focuses more heavily on completion efficiency, reasoning stability, and avoiding pathological long-tail trajectories while retaining the capabilities learned by the original Flash model.

Python formatting was another major target

One of the most useful outcomes of community testing on the first Flash release was the discovery of a Python indentation failure mode.

Certain generations could produce structurally correct Python logic while using malformed indentation, particularly in nested blocks.

This was important because coding benchmarks that focus mainly on semantic correctness may fail to expose formatting problems that make generated code impossible to execute.

The investigation suggested a possible training-side explanation.

Some reconstructed reasoning traces in the training mixture may have contained malformed Python formatting. If a verifier rewarded the semantic trajectory without reliably rejecting the formatting defect, reinforcement learning could unintentionally strengthen that behavior.

V2 was therefore developed with this failure mode specifically in mind.

In the reported local Python-formatting evaluation, the issue was substantially reduced under the recommended coding configuration.

The current recommendation for coding and agentic software-engineering workloads is:

enable_thinking = True
reasoning_effort = "xhigh"
temperature = 0.3
top_p = 0.95
top_k = 20

An interesting observation from local testing was that reducing the coding temperature to 0.3 did not simply stabilize formatting — it also reduced average reasoning length compared with a higher-temperature configuration.

For coding workloads, this combination appears to provide a better balance between deliberate reasoning and deterministic output structure.

These are still local results rather than a guarantee that every Python generation will be correctly formatted, but they provide a useful deployment starting point.

Capability was not sacrificed simply to shorten reasoning

The Flash objective has never been “shortest answer wins.”

A shorter reasoning trace only matters if the task is still solved correctly.

For the V2 evaluation, 2,500 questions were sampled from MMLU-Pro rather than running the complete benchmark.

On this 2,500-question subset, Qwopus3.8-27B-Flash-V2 achieved 92.33% accuracy, compared with 91.28% for the earlier Qwopus Flash evaluation and 92.73% for the reported Qwen3.8 base result.

Because this evaluation uses a sampled subset rather than the complete MMLU-Pro test set, and because the compared results were obtained under different evaluation configurations, these numbers should not be interpreted as a perfectly controlled full-benchmark A/B comparison.

What they do suggest is that the second round of post-training did not require collapsing the model's general reasoning capability simply to make inference more efficient.

The existing agentic software-engineering evaluation also remains an important part of the project: the reported 14-task hidden-test battery was completed successfully, with tasks evaluated through deterministic verifiers rather than an LLM judge.

The broader idea behind V2

Qwen3.8's reasoning-effort system points toward an increasingly important direction for local reasoning models.

The question is no longer only:

How intelligent is the model?

It is also:

How much reasoning should the model spend on this particular task?

For a local model running inside an agent, IDE, research pipeline, or autonomous coding loop, inference efficiency becomes part of model capability.

A model that reaches the same correct result with less unnecessary deliberation can often complete an entire workflow faster, use less GPU time, and support more concurrent workloads.

Qwopus3.8-27B-Flash-V2 is another experiment in that direction:

strong reasoning when it is needed, less wasted reasoning when it is not.

The model is available on Hugging Face as:

Qwopus3.8-27B-Flash-V2-GGUF

with GGUF builds also available for local inference.

Feedback from different hardware, quantizations, reasoning-effort levels, coding workloads, and agent frameworks is very welcome.