RECAP: The megakernel is a CUDA engine for Qwen3.8-27B that runs 1.4-1.9x faster than llama.cpp on a single 3090. The most common questions were about accuracy, the baseline and context size, so here are the numbers.
Log-loss vs the reference: +0.0005 +/- 0.0004 at 1K, +0.0018 +/- 0.001 at 30K, +0.0012 +/- 0.001 at 96K. Not significant.
The 30K and 96K rows are one sequence of code each & the 1K row is code and prose.
Speculative decoding is exact: greedy output is the main model's greedy choice, and sampling keeps the same distribution as without drafts.
Speed (same 3090, megakernel vs llama.cpp with MTP)
Writing code: 140 vs 73 tok/s
Reasoning mode: 78 vs 51 tok/s
With 30K tokens of code in context: 73 vs 52 tok/s
Prompt processing (prefill): ~1,600 vs ~1,100 tok/s
With speculative decoding off they are about equal (42.9 vs 40.4). The gain comes from checking 4-5 drafted tokens per pass for about the cost of one. llama.cpp ran at 2 drafts, the megakernel at 4.
Since the first post
Two PRs have been merged: the loader now checks every tensor's type, and the server warms up before accepting requests.
Model download fixed after unsloth removed the Q4_K_M from their repo.
It also loads the Q4_K_M from lmstudio-community and mradermacher.
openjet setup shows which engine each model will use.
Quantisation support
This table details the quantisation support, with unsloth Q4_K_M being the native model and what I benchmarked, lmstudio and mradermacher being semi-compatible, not fully benchmarked, tentative results show similar speed to native after type conversions. Other quants fall back to llama.cpp
Qwen3.8-27B file
Runs on
unsloth's original Q4_K_M (what openjet setup downloads)
megakernel
Q4_K_M from lmstudio-community, mradermacher
megakernel, semi-compatible
Other quants (unsloth UD-*, bartowski, GSQ-RCO and others)
llama.cpp
Limits
Single GPU. RTX 30/40/50 series with 24 GB or more. MK is tuned on a 3090 only.
147K context runs on a 3090 (fp16 KV cache to ~80K, using q8 above).
No task-level pass-rate benchmark yet.
Contributions that would help
Tunes for specific hardware: RTX 4090, 5090 and other 24 GB+ cards. It has only been tuned on a 3090.
Support for other quantisations of Qwen3.8-27B.
Please include benchmarks with the PR: your GPU, and bench/bench.py or bench/code_bench.py numbers before and after.
How to contribute: see CONTRIBUTING.md in the repo.
Comment the quant you use for Qwen3.8-27B so I know what to add support for next! (also planning for Qwen 4 when that drops).
Numbers are very nice, cheers! One of my GPU id 3090, but, unfortunately, for context, I use two GPUs , and all these custom speedy forks pass by my, only llamacpp is flexible enough to utilize it. :(
Same boat, so I made a two-GPU fork of this one. On 2x 3090 it does 164 tok/s average vs 100 for llama.cpp with tensor split, and it holds up at long context (105 vs 67 tok/s with 56K tokens of code in the prompt). Catch: both cards need PCIe P2P, which on GeForce means the patched open driver.
= not measured (prompt was already cached from the C1 run)
3080 Ti Laptop - GInfer v1.0.0.3, KV reserve 256 MiB - full pass (tok/s)
Cell TG Cold PP Cold TTFT Warm PP
Qwen smol C8, 2048 in/4000 out 378.2 618.7 3.3 s 776,000
Muse smol C8, 2048 in/4000 out 289.8 623.3 3.3 s 758,000
Qwen smol C1, 162,240 in/512 out 52.7 324.9 499.5 s 9,770,000
Muse smol C1, 129,984 in/512 out 58.4 432.0 301.0 s 17,600,000
Faster than any other inference engine by a lot. Support Qwen 27B / Muse 30B, Qwen flash next + GLM 5.3 Flash, tiered offloads, sliding attention, state of the art K/V system, adaptive draft lengths on Dflash2, adaptive prefix caching. It is slick. Can't wait for them to release it (should be in a week or 2).
what is qwen smol ? I'm sorry but 27b @ 269 tok/s tg on a single stream with a 512gb/s card is either very low quant and dflash 100% acceptable (not realistic) or i can't believe it. But maybe i'm wrong and it's insane ? Can you run some real output like try to output 20k token on a real prompt and mesure speeds ? I'm curious.
Smol is made for 16GB cards (like the laptop), it is a mixed precision 2/4/8 bit quant, with some left in BF16/FP32. The test above is a kernel tester, it is a 2k repeated prompt pattern designed to test the kernels during tuning, PP, and the prefix cache.
Not sure if you noticed, but the very bottom block has 2k in, 4k out, and 162k in, 512 out.
3080 Ti Laptop - GInfer v1.0.0.3, KV reserve 256 MiB - full pass (tok/s)
Cell TG Cold PP Cold TTFT Warm PP
Qwen smol C8, 2048 in/4000 out 378.2 618.7 3.3 s 776,000
Muse smol C8, 2048 in/4000 out 289.8 623.3 3.3 s 758,000
Qwen smol C1, 162,240 in/512 out 52.7 324.9 499.5 s 9,770,000
Muse smol C1, 129,984 in/512 out 58.4 432.0 301.0 s 17,600,000
Sure, give me a few and I will run it and post it here.
It is a kernel test, it is repeating prompt pattern that hits all the different kernels. Tests the inference kernels on each layer, and the cold > warm PP test the prefix cache and K/V system. It is designed to test inference against the full hardware capability to look for areas that can be improved. each model / GPU combo has it's own hot path and kernels.
Supports 4 models right now, Qwen 27b, Muse 30B, Qwen next flash, and GLM 5.3 Flash. Each with 1/2/4 GPU's, C1-C8.
Supports hybrid inference, multi-stage vram > memory > disk without skipping experts like Strata / infernix, has CPU kernels for AVX, AVX2, AVX 512, AMX, VINNI, etc. will do CPU only inference as well. Windows, WSL, and Linux.
Supports all A100/CMP 170HX (sm_80), 30xx, 40xx, 50xx, and all RTX pro Blackwell.
Not sure when they are going to release it, Not my project, I just know them and I am just testing. Told a week or two, working on the kernels for Qwen flash next right now.
Also has a pretty slick little server manager in it, all the hosts on your LAN, register and pair, you can define work pools. Single chat client and interface can and control all hosts, and distribute swarms across the work pools, or you can use the dedicated server manager.
You need to document where that inference speed could possibly come from and what interventions had which impact. Because this is well beyond what that card could possibly give with that model.
That is what it is running, test is a 2k repeating prompt pattern used to test the different kernels, (which is why the warm PP is so high, it is 100% cached after first read) + dflash2.
Awesome work. I had Claude and ChatGPT have a go at similar optimisations the other week but yours blew mine out of the water. The things I tried was n-gram reuse, tuning DFlash, some kernel tuning and what I called "card shaped quant", that is quantising such that the weights fit the card's quirks. I ran a head-to-head test of your version against mine (on the exact same weights, yours) and your megakernel approach is slightly better.
So I think for now one approach to improve your engine is to fiddle with the weights themselves a little bit -- that gave me about 15% boost using stock kernels. I tried bringing my n-gram reuse approach to your engine and while it massively increased code edit tasks (decode boost well over 100%) it hit prose, coding and thinking.
Besides fiddling with the quants, I found that you can offload the mmproj weights to regular RAM and pull them JIT for image encode. For single images it's barely a hit on processing time.
Comparing to what the 3090 could theoretically achieve there is still some room to grow:
Path
Current, RTX 3090
Ideal hardware ceiling, RTX 3090
Realised efficiency
Prompt processing, short context
1,595 tok/s
~2,915 tok/s
~55%
Decode, no drafting
42.9 tok/s
~58 tok/s
~74%
Have you considered the same megakernel approach with other models like Qwen 3.6 35B-A3B or Nemotron Lightning 30B-A3B?
I made a tensor-parallel fork of this. On 2x 3090 it does 214 tok/s on code and 164 on average, vs 128 / 100 for llama.cpp tensor split on the same cards, and 105 vs 67 tok/s with 56K tokens of code in context.
Ran your KL test on it: the original, the fork on one card and the fork on two cards all score the same (96.75% top-token agreement, KL 0.004), so splitting costs no quality.
On more quants: the fork loads any per-tensor mix of Q4_K, Q5_K, Q6_K and Q8_0. Only Q4_K_M is benchmarked so far, but happy to validate the others and send it upstream as a PR if you want it.
13
u/Barni275 4d ago
Numbers are very nice, cheers! One of my GPU id 3090, but, unfortunately, for context, I use two GPUs , and all these custom speedy forks pass by my, only llamacpp is flexible enough to utilize it. :(