r/LocalLLaMA • u/Adorable_Weakness_39 • 5d ago
Resources Qwen3.8-27B on a single 3090: 140 tok/s on code with a custom megakernel
I've been using Claude Opus 5.5 to speed up Qwen3.8-27B on my PC (rtx 3090), it wrote a CUDA megakernel that is 1.4-1.9x faster than llama.cpp depending on the task/context length. The results and code are below:
Results (same hardware, my megakernel vs llama.cpp with MTP):
- Writing code: 140 tok/s vs 73 (1.9x faster)
- Coding with a 4K-token file in context: 95 tok/s vs 56 (1.7x faster)
- Prompt processing (prefill): ~1,600 tok/s vs ~1,100 (1.4x faster)
The speedup comes from running each whole speculative-decoding cycle in one kernel launch, so checking 4-5 drafted tokens costs about the same as checking one. It's an OpenAI-compatible server, so it works as a drop-in replacement for the llama server.
Caveats and limitations: it's built for one model (unsloth's Q4_K_M) on 3090s only. The outputs match llama.cpp's apart from rare rounding near-ties. KVCache is fp16 up to ~80k context, then q8.
Code and full benchmarks: https://github.com/L-Forster/open-jet/tree/master/megakernel
21
34
u/CoffeeToCode99 5d ago
Love seeing optimisation-focused work like this. The AI space has become so obsessed with bigger models that people forget software still matters.
15
u/datbackup 5d ago
Very impressive! If there’s any chance you could do another megakernel for https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF IQ3_S i think it would be very well received. It would free up another 4ish GB VRAM for kv cache!
6
u/Adorable_Weakness_39 5d ago
I will have a look, how does the KL divergence compare with the unsloth Q4_K_M?
2
u/datbackup 5d ago
IQ3_S mean KLD is 0.048 according to this repo:
https://huggingface.co/Joakimpalm-Zen/Qwen3.8-27B-GSQ-RCO-IQ3_S-recovered-GGUF
Original repo i first linked doesn’t include KLD that I could see. They show perplexity and task-exactness compared with BF16.
10
u/zyxciss llama.cpp 5d ago
Is output byte identical?
6
u/Adorable_Weakness_39 5d ago
Not byte-identical. It's the same model, but the rounding is done differently so when two tokens are nearly tied it can pick the other one.
15
3
u/valarauca14 5d ago
Top K agreement? KL divergence? cosine similarity?
1
-2
5d ago
[deleted]
1
u/valarauca14 5d ago
KL-Divergence is how to compare two probability distributions. Quants use it because it is a short hand for, "How much does this model differ". Because it is an objective measure of that. Literally how does the probability distribution change for the same quant.
When people merge/tinker with kernels you use KLD as a way to measure, "How much did I fucked up the math" and "Did I accidentally change the model's behavior" or "KLD = 1e-11 everything is fine".
1
u/Ctbhatia 5d ago
cheap check is greedy decode with a fixed seed on both kernels and a diff of the token ids, small drift is expected since kernel order changes fp accumulation.
7
u/vexter0944 5d ago
The box
OS: Ubuntu 22.04.5 on bare metal
Hardware: Ryzen 5 5600X (6 cores / 12 threads), 31 GB RAM, RTX 3090 24 GB.
GPU stack: driver 580.178.04 with CUDA 13.
So far looks like a solid improvement - will test drive and see how it goes.
1
u/Distinct-Pie2389 5d ago
Did you try ninfer-3090 before this? Can you compare please
3
u/vexter0944 5d ago
No the only thing I had going on was club 3090 - nothing else. I'll try to find some time to look at the ninfer-3090 and if I do, I'll report back.
2
u/Distinct-Pie2389 5d ago
Oh I see, Im going to steer away from this from now. Llama.cpp measured baseline was weak on this, looks like --spec-draft-n-max 2 baseline and then jumped MTP 4 on megakernel, ceiling 100k context, and fp16 accumulation so I’m going to avoid this it’s a nice engine but it’s not going to beat ninfer build
1
u/GrungeWerX 4d ago edited 4d ago
What is ninfer and what’s been your experience using it?
2
u/Distinct-Pie2389 4d ago
ninfer-4090 is a C++/Cuda engine designed specifically for Qwen3.8-27B on NVFP4 but was converted to work on non Blackwell cards like 3090 and 4090
1
u/Goliath_369 4d ago
2
u/GrungeWerX 4d ago
Maybe I was interested in his opinion since he uses it rather than a generic ai explanation.
15
u/bugra_sa 5d ago
140 tok/s is wild, but I'd love to see the number again at a long context. That's where the KV cache can change the story. Posting the exact command and power draw would make it much easier for other 3090 owners to sanity-check.
4
u/Adorable_Weakness_39 5d ago
Testing with your devices and PRs with tuned kernels for your GPUs are welcome <3
5
u/simi6a6 5d ago
Does it support dual/multi RTX3090 or only single card supported?
2
u/OhGeeLIVE 3d ago
It does now. I forked it and made it tensor-parallel across two cards: 214 tok/s on code and 164 on average on 2x 3090 (280 W each, no NVLink), vs 128 / 100 for llama.cpp tensor split on the same cards. Same KL/top-token quality as the single-card version.
Needs PCIe P2P between the cards (on 3090s that's the P2P-patched open driver). https://github.com/ohgeebtw/dual-gpu-megakernel
Cant post a thread about it because of low karma :) Will do as soon as I can
1
3
u/caetydid llama.cpp 5d ago
whats your use case on decode speed optimization? my harness has 4/5 prefill and 1/5 decode tokens, so I am solely interested in speeding of prefill tps. I cant find numbers for those in your repo, is it not important for you?
3
3
u/OhGeeLIVE 3d ago
Great work on this. Since a lot of people here asked about dual 3090s: I forked it and made it tensor-parallel across two cards.
2x 3090 (280 W each, PCIe 4.0 x8, no NVLink), Q4_K_M, tok/s:
| Test | 1x 3090, this megakernel | 2x 3090, llama.cpp tensor split | 2x 3090, fork |
|---|---|---|---|
| Writing code | 140 | 128 | 214 |
| Chat prompts, average | 107 | 100 | 164 |
| 56K tokens of code in context | n/a | 67 | 105 |
Same quality as the single-card version on OP's KL test (96.75% top-token agreement, KL 0.004, same perplexity). Needs PCIe P2P between the cards, which on 3090s means the P2P-patched open driver.
2
2
2
u/SnooPaintings8639 5d ago
What's the max context a single 3090 with this code can hold? What is speed for prose and short prompts?
And, if anyone with more than one 3090 could try... Is it better to run two independent instances or a single instance on two cards?
1
u/OhGeeLIVE 3d ago
Measured this on 2x 3090 with a two-GPU fork of this megakernel: one instance split across both cards does ~164 tok/s for a single stream (the second card adds about 54%), plus you get double the KV memory for long context. Two separate single-card instances give two streams at ~107 each, so more total throughput if you actually have two parallel users. For one person coding, the split wins.
2
u/caetydid llama.cpp 5d ago
does openjet allow running remotely? I have my gpus on a dedicated machine, but I like to do the coding on my laptop
1
u/Adorable_Weakness_39 4d ago
It is an OpenAI-compatible server with --host and --port flags, so it can run on the GPU machine. Then you can use any harness or OpenJet's adapted Pi harness with it.
2
1
u/ApprehensiveAd3629 5d ago
what are the tdp of your 3090 during these tests? i got around 30-35 tokens/sec with 220w as max but it goes up to 40-45 tokens/sec when using 250w
1
u/keepthepace 5d ago
I wish I had time to test all the cool toys raining on us these days! Thanks! Will had it to the pile!
1
u/Sugisaky 5d ago
I do wonder how this will compare to Hyperqwen that gives me around 133 tk/s, will check it out later, also how big is the context window?
1
1
u/Civil_Fee_7862 4d ago edited 4d ago
Nice work.
Any chance it could be modified to work on dual RTX 3090s?
1
u/FlyingPoptartz 4d ago
how does this compare to hyperqwen and can it be used on the qwen finetunes that optimize overthinking
1
1
1
0
u/vogelvogelvogelvogel 5d ago
will it work on a 4090
2
u/Distinct-Pie2389 5d ago
I already A/B tested vs ninfer-4090. I would check out both and make your own decision! Both great engines, you’ll need to tweak OP’s code to get 262k context on the 4090 or you’re preferred as you’ll have a little more headroom on sm_89 via bandwidth.
28
u/Mayion 5d ago
cant wait for gigakernel