Sampling: temp 1.0, top_p 0.95, top_k 20 for both. Run with evalscope over the OpenAI-compatible API: Qwen3.8-Next on Strata (MI50 32GB), Qwen3.8-27B on llama.cpp (RX 7900 XTX 24GB).

Caveat: These are only a portion of the full sets, since they take days.

Edit1: Strata started with --kv fp16 and llama's default f16.

Results

Sign test on the 14 problems only one model solved (10 vs 4): two-sided p = 0.18, so a lean toward Next rather than proof.

Token limits

Failures: wrong answer vs out of tokens

BFCL v4 by category (7 per category)

Speed (AIME + LiveCodeBench generation)

Hey everyone, I’m trying to decide what to do with my local LLM setup before putting more money into it.

A bit of background: I’m a final-year electrical engineering student, and I use local models for learning about LLMs, experimenting stuff and my thesis on financial analysis of solar plants with BESS. I also am going to work in strategic consulting after graduation, so I want a setup that is useful for data and market analysis and some coding.

My current machine:

• i7-7800X
• MSI X299 Raider
• 128GB RAM — 8×16GB DDR4 2666MHz
• 2× RTX 3090

I’m currently running Strata IQ3_S at around 100 tokens/s, and I’m pretty happy with the speed. Quality seems okay, a bit better than 27B at Q8.

I also have five more 3090s: one ASUS ROG Strix and four Zotacs. All five need repasting and new thermal pads. So seven cards total, with only two currently in use.

I paid about $900 each for the four Zotacs. That’s part of why I’m torn: I could use them for a bigger setup, but I could also sell them for around $1,300 each pretty easily in my local used market.

These are the options I’m considering:

1. Upgrade the existing X299 machine

I found an i9-10940X on AliExpress for around $300. It would give me more cores for my other server workloads and more PCIe lanes, without replacing the motherboard and RAM.

My understanding is that I could then run the GPUs at x16/x16, depending on the motherboard’s slot wiring. Please correct me if there’s a limitation I’m overlooking.

Alternatively, I could use more widely spaced slots at x16/x8. I’ve tested that layout, and the top GPU runs about 10°C cooler, with the cards sitting around 65°C and 58°C.

Would you spend $300 on this CPU upgrade, or avoid putting more money into X299? For inference, would you prioritize x16/x16 or the better cooling at x16/x8?

2. Build a separate machine for the four Zotacs

This would give me a second machine with 96GB of VRAM. I’d need to buy a suitable CPU/motherboard setup and sort out cooling, power, and the rest of the build.

For anyone who has gone from 48GB to 96GB:

• Was the improvement in model quality worth it? Im looking at maya for GLM 5.3 flash.
• Which models and quantizations made the upgrade worthwhile?
• What inference speeds are you getting, and with which backend?
• What reasonably priced platform would you recommend for four 3090s? Used hardware is fine.

I’m happy with my current speed. The main reason to expand would be access to better models or higher-quality quantizations for engineering, financial modeling, document analysis, and coding.

3. Sell the Zotacs and keep the current setup

Since I paid about $900 per card and could sell them for around $1,300 each, I could also sell them and avoid the cost and work of another build.

Would you keep the cards because 96GB opens up enough useful possibilities, or sell them while the local resale price is still decent?

This is an offline app with 51 hands-on labs for learning how AI actually works, from neurons to agents and more...

**A bit of insight**

- 51 labs in 6 groups, from the basics (what a neuron is, gradient descent) through attention, RAG, agents, fine-tuning, quantization, serving and more

- Each lab has a short lesson beside it, readable in Plain or Standard mode

- Some of it actually runs rather than just animating:

- the attention lab runs a small trained transformer (1.37M params) inside the app

- the training labs train a tiny character-level model live as you move the sliders

- Works offline: no account, no telemetry

**How it was made**

I chose the topics, some background research and came up with the structure and the grouping. Claude and GPT Astra to help write the lesson text, and Grok as a second pass on references. There will be mistakes, so if you spot one, please tell me or open a PR.

If you teach this or work in a specialized area of AI, I'd love help expanding it: a new interactive lab, a better visualization, or a tweak that makes the cause and effect in an existing lab clearer. I'd also like to hear which labs are confusing and what's missing.

GitHub: https://github.com/Fazmin/AILearningGuide

I've been working on an inference engine called jitllm. The idea came from my background in operating systems and JIT compilers: generate kernels for whatever hardware is there, and treat layers a bit like processes the engine can place and move around.

In the video, a cloud VM with T4s: mid-reply, all layers of Llama-3.2-3B move from one GPU to the other and back. And my puny laptop where the model is split across CPU, CUDA and Vulkan (iGPU), rearranged while it's generating. The KV cache is paged and moves with the layers, so the prompt isn't re-processed and the output matches a run that didn't move.

What excites me is the room the design leaves. Generated kernels mean every improvement reaches every machine, and the engine can fuse and tune for hardware it has never seen. Paging and relocation turn placement into scheduling: models bigger than memory, mismatched GPUs, work that shifts as conversations grow.

Code -> https://github.com/jitllm/jitllm

I spent some time taking best practices from the past two years developing my projects with agent skills and centralized them in a repo so I can reuse them and standardize my projects. In the process of doing that, I realized that skills exist in different scopes and timelines, so I spent some time trying to figure out how to organize skills usefully. I realized that Magic The Gathering abilities have a lot of parallels to agent skills, so I did some experiments separating skills into three categories that MTG uses: activated, static, and triggered. I found this works very well for communicating to agents considerable context about the skill scope with minimal verbosity.

If you're not familiar with MTG abilities, firstly MTG is a complex card game, and secondly each card in the game has some text at the bottom which amounts to a special coding language. The coding language's main block is called an ability, which represents some effect in the game. Abilities are separated into several types which set the scope and timing of the effect. These ability types are directly transferable to skills:

• Activated Skills: These are skills the agent must choose to use, and are a set of procedures that should be performed in order once
• Static Skills (called "passive" in my repo): These are skills which set general rules that should be applied all the time given the agent is working in a particular domain.
• Triggered Skills: These are skills which are like activated skills, but they should always happen any time a skill-defined event occurs. The event can be anything: something the agent does or an external event like a time passing.

Separating the skill into these categories lets the agent have some preconceived ideas about how the skill should be used, which increases reliability and decreases context redundancy.

I thought this was an interesting notion and just wanted to share. If you check out my skill repo, it has some other good ideas I've collected over time like a hermes-inspired skill self improvement system, test process organization, and documentation style/methodology. If you have any ideas about making it better, let me know.

We just shipped web search in Private Mind, our open source app for running LLMs locally on iOS and Android. The video is the whole pitch: Qwen 3 1.7B with web off makes up Apple's September event, with web on it gets it right and cites 5 pages.

The fun constraint is that a 1.7B model with a short context can't just read 5 web pages. So most of the work is picking the few hundred words it actually gets to see.

Roughly how it works:

• There's no search API. A hidden WebView on the phone opens DuckDuckGo (Brave and Mojeek as fallbacks) and we parse the results page
• It opens the top pages and cuts them into ~500 character chunks
• An on-device embedding model (LFM2.5, ~415MB) plus some keyword matching picks the best 6 chunks, max 3 from one page
• If only one site claims something, it counts for less than something two sites agree on
• The model answers from those chunks and links every source

Web search is a per-chat toggle and off by default. When it's on, the phone makes the same requests a browser would. The chat itself stays on the device.

Everything runs on react-native-executorch, both the LLM and the embeddings.

Repo: https://github.com/software-mansion-labs/private-mind

Inspired by how short-term memory works in our own brain, I built PulsarLM, an inference engine that removes old and unimportant tokens from the context window to keep it bounded while also allowing the conversation to go way past the trained limits.

Running Qwen3 8b on a 5090, I get 6.7k tok/s during prefill and 89 tok/s during decode. It does support batched decode and with 16 concurrent requests, I get 1.2k tok/s.

It works by using custom attention kernels that accumulates an attention mass per token page. At regular intervals, pages that receive low attention mass gets removed.

To workaround out-of-distribution positions, embeddings are re-rotated (for RoPE) such that token positions stay contiguous and never exceed the context limit.

Eviction of tokens in the context window would also break prefix caches normally, but PulsarLM also caches snapshots of context to avoid unnecessary re-prefills, but does use up extra RAM. There is also an option to persist those snapshots.

Only a smaller and older models are implemented for now, and the engine itself still has rough edges. It's only been tested on linux x86_64. Feedbacks are welcome! Also, what models would you be interested to try?

Here's the link: https://github.com/cosmonomic/pulsarlm

Edit: There used to be a recall mechanism using bm25+attention score, but removed it since it didn't make sense to store that many KVs. So right now, it works best when paired with an external memory system/RAG.

And also, running benchmarks right now, will update once results are out.

a new option to optimize your VRAM

[UPDATE / RETRACTION: The "100% GTO" claim had a prompt-leakage bug] We patched the repo to pass strictly neutral states, here are the real, honest numbers: • Latency: Still ~2ms (ONNX + Absorbing Markov Chain). • True Zero-Shot GTO Parity: ~15% - 29% across all 340 canonical hands.

**[Benefits of Tier 0 Absorbing Markov Chain + Embedding model] on JevBench, Tier 0 achieved 12 / 12 early exits with 100% accuracy at 16.5 ms p50, and on Decision Index 0.3, it safely rejected 344 / 344 out-of-domain items with 0 false exits

Gevva0 Cascading Decision Engine:

Tier -1 · Static Rules (<50 µs): Hot-reloadable regex and keyword guards that run in microseconds without touching the GPU

Tier 0 · Absorbing Markov Chains (<3 ms): Embeds queries into a 768D manifold using EmbeddingGemma-2 (740M) and solves random-walk absorption probabilities in closed form, hitting 100% accuracy on early exits with zero false exits on 344 out-of-domain items.

Tier 1 · Fast-Path Logits (~35 ms): Runs a single forward pass on Gemma 4 26B-A4B (MoE, 4B active) with a custom K=64 micro-head that slices the 262,144-row vocabulary down to just the 52 answer letters cutting output-head compute by 99.98% and resolving 77.9% of queries without generating a single token.

Tier 2 · Hot-KV Bounded CoT (≤ 256 tokens): Escalates the hardest 17.1% of requests into bounded Chain-of-Thought reasoning directly from the KV cache Tier 1 already prefilled (reuse_kv=True), eliminating double prompt processing and cutting long-context latency by up to 30%.

Highlights:

• JevBench, Gevva0 beats Claude Opus 4.6 Thinking on chance-corrected intelligence (84.62 vs. 83.90) at a 44 ms median latency (vs. Opus's 4,850 ms)
• Decision Index 0.3, our 4B-active MoE cascade outperforms dense 27B models (like Kev 27B) at a 60 ms p50—roughly 7× faster than 27B alternatives.
• 60 FPS Real-Time Game-Loop Ready (Pacman demo included)

https://github.com/solvingSteve/Gevva0

Hi everyone — I’ve been building a local-first AI tool for 2D game animation.

The goal is simple: reduce the tedious parts of character animation work without depending on cloud APIs or losing control over the art pipeline.

It takes a reference image/character setup, reconstructs a usable 2D asset pipeline, handles rigging/retargeting and animation append flow, and exports game-ready output in a format suited for 2D game production.

This is especially useful for:

• indie devs and solo creators
• technical artists
• animators who want faster iteration
• teams that want private local workflows

What makes it different:

• runs locally
• avoids uploading game assets to external services
• targets real workflow bottlenecks, not just image generation
• designed for game production, not “AI magic” output alone

Project page: https://donatelloai.co/

Steam: https://store.steampowered.com/app/5030490/Donatello/

I’m sharing this as a build-in-public project and would love feedback from people working on local AI tools, pipeline tooling, or game animation workflows.

Would be great to hear what people think, especially around workflow quality, artist control, and practical production use.

Are we seeing early signs of the bubble popping?

I'm using Qwen 3.8 27B on an R9700 and it works great, occasionally I'll switch to Tiel-coder for rapid simple tasks. One big problem I run into that makes real projects come to a halt is that more code takes up more context, compaction becomes more and more frequent between any edits, until it gets stuck in a compaction loop not doing any useful work. I started using Q4_K_XL and Q8 kv cache, for a context of 200k tokens, which helps from my normal 100-150k, but is there a technique to ensure it works efficiently?

One idea I have but never fully enforced is having the code indexed (function signatures with comments and line numbers, structs shown in full so it doesn't need a read, maybe methods listed under structs.) Or another way is to make the code extremely modular so the LLM will have a very narrow focus. I tried this but it becomes very cumbersome because other modules will also need changed, probably until each becomes mature enough?

I'm sure some here read this at the time and remember, but I recently found this hilarious Guardian article from 2019 through Wikipedia:

https://www.theguardian.com/technology/2019/feb/14/elon-musk-backed-ai-writes-convincing-news-fiction

Some choice quotes:

"OpenAI ... says its new AI model, called GPT2 is so good and the risk of malicious use so high that it is breaking from its normal practice of releasing the full research to the public in order to allow more time to discuss the ramifications of the technological breakthrough."

"One is its size, says Dario Amodei, OpenAI’s research director. ... It was trained on a dataset containing about 10m articles, selected by trawling the social news site Reddit for links with more than three votes. The vast collection of text weighed in at 40 GB, enough to store about 35,000 copies of Moby Dick."

Some people never change!

Gguf's: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B-GGUF

Free Demo: https://playground.liquid.ai

"LFM2.5-24B-A2B is a 24B-parameter reasoning model with only 2B active parameters, built for fast, private agents on-device. It is the largest and final model in the LFM2.5 family, which now spans two orders of magnitude: from LFM2.5-230M to LFM2.5-24B-A2B.

• Best-in-class agent: Competitive with Qwen3.6-35B-A3B, a model with ~50% more total and active parameters, on agentic and tool-use tasks, scoring 48.09 on LiveClaw versus 38.19.
• Reasoning at MoE speed: Always thinks before it answers. With only 2B active parameters per token, each reasoning token is cheap.
• Efficient inference: Runs up to 1.7× faster than comparable MoEs on-device, with 184 tok/s on an Apple M5 Max and up to 21K output tok/s on a single NVIDIA B300 GPU.

LFM2.5-24B-A2B is a general-purpose text-only reasoning model with the following features:

• Total parameters: 24.0B
• Active parameters: 2.45B
• Number of layers: 40 (30 double-gated short convolution blocks + 10 GQA)
• Experts: 64 per MoE block, top-4 routing
• Training budget: 36 trillion tokens
• Vocabulary size: 128,000
• Context length: 131,072 tokens
• Languages: English, Arabic, Chinese, French, German, Italian, Japanese, Korean, Portuguese, Spanish, Vietnamese, Thai, Indonesian, Hindi, Russian, Polish
• Generation parameters:
  • temperature: 0.1
  • top_k: 50
  • repetition_penalty: 1.05"

I finished putting my Gyro-M experimental setup on GitHub:

https://github.com/Gitshop77/GyroStrata

Original post : https://www.reddit.com/r/LocalLLM/s/4LmgejUQH4

Running full 256K context on a 4070 Super 12GB and 32GB RAM, getting around 950–1800 tok/s prefill and 30–38 tok/s output depending on context. Note that mtp and vision are off on purpose . they were affecting speeds in many test trials. If you have 16vram you could possibly try using MTP.

Includes the Strata patches, tuned settings, model downloader, installer and launcher. MTP and vision are off.

Speeds depend heavily on your CPU, SSD, RAM speed and GPU bandwidth.
Don’t Expect to get similar results Against my results initially Unless you have the exact specs of my system . There’s a prompt u can give to Codex/Claude Code/etc included to tune it for your own specs. All benchmarks and setup details are in the repo.

Don’t expect guarantees of fast speeds . It’s a tight fit, and I’m not sure how you are supposed to send recipes for multi-component stuff . I know in DGX Sparks recipes you can just use the same exact config since it’s the same memory and bandwidth, same goes for dual 3090 recipes and etc . So I provided a prompt for a coding agent so hopefully it tunes it well for great speeds.

I've been tinkering / vibing myself an app that I think might be useful - since I've now used it on my own cases for some time and fixed most of the bugs I noticed, I'm risking a public announcement :)

https://pwilkin.github.io/ROCprofGUI / https://github.com/pwilkin/ROCprofGUI

The ROCprofGUI (aka rocprof studio) is a tool I made to be an alternative to ROCprof Compute Viewer and other tools in the ROCm ecosystem - one that is user-friendly, has a modern UI and can be used with agentic help.

Despite its name, the studio can analyze traces from all of the engines that AMD hardware supports: ROCm traces from rocprofv3, Vulkan traces from RADV (needs a patch to the driver currently) and HRX traces for Loom kernels.

Coupled with the studio is an MCP server that allows tracing from your favorite LLM agent. The studio works in two modes - as a GUI (then, if you use the MCP, a popup appears telling you that the MCP server is currently driving operation) or fully headless - in which case there's no UI and just the service running the MCP server.

I've been running it for over a week now while doing my optimization work on llama.cpp, so hopefully the most annoying bugs have been ironed out. I hope it's of use to someone :)

I'm excited to see our Lemonade embedded SDK getting used out there!

TL;DR: I made a small multi-agent setup for pi. It is one skill and one Python script, with no extension and no tool. Your agent can start subagents in the foreground in parallel or in the background, and it can wait for them, steer them, abort them and resume them. Background jobs keep running after the main pi session is closed or killed. A new pi session—even one different from the original main session—can also retrieve their results. Until your agent decides to delegate, it costs about 50 tokens of context.

Repo: https://github.com/milanglacier/pi-multi-agent-skill

Just install it with npx skills like any other skills, or clone the github repo and symlink it to ~/.pi/agent/skills. Or install it through pi install as it is also packaged as a pi package.

Features

• Almost no token cost. Until your agent delegates, only the skill description is in context, a few dozen tokens. Delegating loads SKILL.md, about 1.2k more.
• Stdlib-only Python 3.9 or later. There's no extension, no npm package and nothing else to install.
• Foreground runs for quick questions and --background for long ones. A background job keeps running even if the main agent is killed, and any later session can collect its result. Foreground runs stop when you stop at every level of nesting.
• steer, abort and resume a job, and list, status and peek to see what it's doing.
• Subagents load your full pi setup: extensions, MCP servers, skills and settings.
• Profiles set the system prompt, model, thinking level and available tools. general-purpose and a read-only Explore are included. Profiles are just like any other subagent tool’s profiles.
• --fork gives the subagent your current conversation.
• Works from Claude Code, Codex or any agent with a shell.

How I got here

For months I used tintinweb's pi-subagents, and it is a really good extension. The UI is nice, it has many features, and it works well most of time. Thanks to tintinweb for building it and for everything they give the community. If you like it, keep using it.

A while ago a pi upgrade broke two things for me. MCP servers stopped loading in subagents, and subagents no longer inherited the parent's thinking level. Each fix was only a few lines, so I forked the extension and patched it. Upstream fixes took some time but also landed 2 days before this post.

Fixing those bugs made me read the extension more closely, and I noticed how much context it now uses. About two months ago it added a dynamic workflow tool. With it, the agent writes a JavaScript script that orchestrates many subagents. It's powerful, but it's also very complex, and its tool schema and instructions cost about 6k tokens. That schema is in every session from the first message, whether you use workflows or not. It's also opt-out, so you pay for it until you find the setting and turn it off. I suspect many people don't know it's there.

With the workflow tool off, the Agent tool still costs about 3k tokens, and it has grown along with the extension's features. I checked these numbers by starting sessions with the extension on and off and comparing them.

So I asked myself whether I need a full tool with a detailed JSON schema, listing options I rarely use (like worktree isolation), in my context all the time. My answer was no.

pi can already run headless with --mode rpc, and its CLI flags already cover what a subagent needs: tools, extensions, model, thinking level and session resume. So a small CLI that starts pi with the right flags, plus a skill that explains how to use it, can replace a dedicated tool. A skill also loads in steps, which a tool cannot do. The agent sees only the skill's short description at first. It loads the main instructions when it decides to delegate, and it reads the detailed references only when it needs an unusual option. Steering doesn't need a tool either, because it is just an RPC message sent to the running pi process.

Token cost compared

• pi-subagents: about 3k tokens for the subgent tools and injected system prompt. On top of that, about 6k for the dynamic workflow tool, which is on by default and must be turned off by hand. Both are in every session from the first message, so with the default settings that's about 9k tokens before you've asked for anything.
• This skill, before your agent delegates: only the skill description, ~50 tokens.
• This skill, once your agent delegates: SKILL.md is loaded, about 1.2k tokens, and that is enough for full multi-agent use.
• Command details, profile format, lifecycle and troubleshooting are in separate reference files. The agent reads them only when it needs them.

For most sessions you don't delegate, you pay almost nothing. When you do delegate, it still costs less than the tool schema alone.

To be fair, this isn't a like-for-like comparison. pi-subagents does more than this skill. Its tool has many options, and each one needs a parameter in the schema, and it also adds text to the system prompt. Part of that token cost pays for features this skill doesn't have, such as worktree isolation. I just don't think features like that are needed for multi-agent work. If I want an agent to work in its own worktree, I tell it so in the prompt ("use worktree"), and it does. The agent already knows how to use git, so the tool doesn't need to do that for it.

None of this is meant as criticism of pi-subagents. It's still a great extension, and it works well for a lot of people, especially if you want a rich UI and many features built in. It's just not the right fit for me anymore. I prefer a more minimal setup, where the agent loads only what it needs and does the rest itself.

The tradeoff

I give up the nice UI. It really does look nice, but for me it turned out to be like the exercise bike you buy in January. It looks great in the corner of the room, and by March it is where you hang your clothes. I watched the subagent transcripts for the first week, and after that I almost never read them. I just realized the subagent's final answer is mostly for the main agent, not for me.

In return, I get things I care about more. Each subagent is a real pi process started with CLI flags, so it loads the same extensions, MCP servers, skills and settings as my normal pi. The script doesn't use pi's extension API at all, only CLI flags and RPC, which change much less often, so pi upgrades rarely affect it. Separate processes also give me my favorite feature: a background job keeps going even if the main agent is killed, and any new session can collect its result later.

Separate processes need more care than in-process sessions,, and the script is more than a cli wrapper, because it takes care of that so the agent doesn't have to. For example, pi exits on its first failed write to stdout, before its shutdown runs, and leaves the agent's commands running. So each job has its own runner process that keeps reading pi's output until pi exits. The script has a real test suite for cases like this, but the idea stays small: start pi, give it a prompt, pass along steer and abort messages, and save each job's result under its name.

Non Goals

These are deliberate non-goals, to keep the skill small and free of bloat. Each is better as a separate extension or skill that you pick to match your own workflow. This follows the minimalism of pi itself and the Unix philosophy: do one thing well and compose.

• Memory. Agents do not share or keep memory between jobs. Use a memory extension, or have agents write handoff notes.
• Scheduling. There are no cron-style or recurring agents. Use cron, systemd timers or a scheduler extension to run the commands.
• Worktrees. Agents are not isolated in git worktrees. Just prompt the agent to use a worktree.
• Dynamic workflows. There is no scripting layer for orchestration such as pipelines, parallel fan-out with gates, or phases. Your agent can already start, wait for and chain jobs with plain commands. For deterministic scripts, use an extension such as pi-dynamic-workflow.

What the commands look like

Your agent runs these by itself, but you can also run them by hand from a terminal:

```python

Ask a read-only agent a quick question and wait for the answer

multi_agent.py run --profile Explore "Where is the retry policy defined?"

Start a review in the background, then collect the result later

multi_agent.py run --background --name review "Review the error handling in src/server/"
multi_agent.py wait review --timeout 120

Correct the running job, then give it a follow-up task when it is done

multi_agent.py steer review "Skip generated files"
multi_agent.py resume review "Now check the client side"

See all jobs and what they are doing

multi_agent.py list ```

If you try it, I'd like to hear what works and what breaks.

This continues the small-model browser experiments from the previous posts.

Two standard, off-the-shelf models ran locally on the same Samsung Galaxy S21:

• Gemma 4 E2B, Q3_K_M as the reasoner
• Qwen3.5-0.8B, Q4_K_M as the executor

No fine-tuning, no task-specific training, no adapters.

A persistent ledger carried task state, constraints, verification and recovery. Browser perception and actions came through E2LLM/SiFR, developed by us, inside the user’s existing Firefox session.

Task: prepare an Instax Mini 12 kit on Amazon under $200.

Run 1: bundle route, 2 cart entries, $190.98.
Run 2: separate products, budget violation on a $179.95 film choice, recovery to a cheaper alternative, 4 cart entries, $143.86.

Both runs ended READY_FOR_HUMAN_REVIEW. No checkout or purchase.

The attached video shows both full runs at x16. Original session: about 51 minutes.

The ledger owns the task. Each model owns only the next bounded decision.

All evidence is public on GitHub in e2llm/edge-browser-agent, folder amazon-dual-slm-ledger: full raw log, exact script, launcher and SHA-256 hashes.

Thoughts, objections, failure modes and alternative designs are welcome.

For anyone interested, I've been doing some kernel and tuning optimization for AMD because it was performing abysmal out of the box. But the effort was worth it as I was able to more than 2x its performance out of the box (the card is extremely new and there are almost no GEMM table and supports yet). (due to both using separate models and environmental factors input token can have ~3% less tokens for AMD to decode despite effort to keep them as close as possible). Just to keep things short and concise, I did a varying tests of small input c=8, medium input c=20, and long input c=1. Everything was done in cold state (no kv-cache at start).

And I think there is more performance to unlock for MI350P as below benchmark was performed when I was only able to unlock 50% mem utilization (currently inference is still heavily bounded by compute because of missing GEMM tuning config)

Model used:
AMD: amd/Qwen3.8-27B-Quark-AWQ-MXFP4
Nvidia: unsloth/Qwen3.8-27B-NVFP4

(Small) 3k-input, 8 user comparison, 100 prompts
AMD:
Total input tokens: 288653
Total generated tokens: 75000
Nvidia:
Total input tokens: 300000
Total generated tokens: 75000

(Medium) 50k-input, 20 user comparison, 20 prompts

AMD:
Total input tokens: 965153
Total generated tokens: 10000
Nvidia:
Total input tokens: 1000000
Total generated tokens: 10000

(Large) 200k-input, 1 user comparison, 5 prompts
AMD
Total input tokens: 962033
Total generated tokens: 25000
NVIDIA
Total input tokens: 2000000
Total generated tokens: 25000

We maintain an updated list of official MCP servers repo: 280+ well-known and widely used projects, all from the companies behind the products or actively maintained open source. Feedback and PRs welcome.

github.com/VoltAgent/official-mcp-servers

"--expert-cache",

"2000",

"--prefill",

"auto",

"--spec",

"2",

"--max-context",

"176128",

"--kv",

"q4_0",

"--resident-experts",

"--kv-resident",

"20480",

"--pcie-frac",

"0.00",

"--spec-min-p",

"0.70",

"--adapt-every",

"1",

"--adapt-swaps",

"80",

"--adapt-decay",

"0.97",

"--vram-reserve-mib",

"300"

the tweak that worked to fix prefill is sacrificing mtp which had very little impact for me and caching 2000 experts. best model I have ever run on this hardware. no tool call issues no looping. even run past 3 compactions when I was doing 192k context but for that you will have to cache less experts. I have run into no issue so if you will use you do your own tests and judge. 3070 and 32gb ddr4