Meta demonstrated, in their Breaking the Token Ceiling: Distilling Smaller, Stronger Byte Models paper, by converting tokens from teacher models into byte models.

I look forward to seeing this in practice.

I'm a sysops/devops engineer at a hosting company and wanted a local model for daily work (Terraform, Cloudflare, AWS, log digging, small scripts, mail and ticket triage). Generic benchmarks didn't tell me much about my work, so I wrote my own: 28 one-shot tasks with deterministic grading. Over 6 days I ran it against my local setup, a handful of cloud models and a dozen config variants. Sharing in case it helps people with a similar box. A lot of the usual advice turned out not to apply to a 2-bit quant.

TL;DR

• IQ2_XS on Strata scores on par with full Qwen3.8 Flash from the API: 47–50/56 locally vs 48–49/56 on OpenRouter/Alibaba, with about half the wall-clock time (median latency 12 s vs 22 s).
• Greedy (Strata's default) is clearly the best for this quant. Qwen's recommended sampling (temp 1.0, top_p 0.95, top_k 20) dropped me from 50 to 37/56. temp 0.6 gave 43, and temp 1.0 + min_p 0.05 gave 40.
• reasoning_effort: high is worse than medium (42 vs 50/56) and twice as slow. On IQ2 the model loops in its thinking. A 16k reasoning budget didn't help either.
• ./setup.sh --calibrate on Strata 0.1.43 gave +25% decode (median 118 → 143–147 tok/s). The gain came from --pool-workers 8 and the adaptive expert tier, not from --pcie-frac.
• Run-to-run noise is about ±3 points out of 56, even with greedy. Check which tasks flipped before concluding anything.

Hardware / software

• RTX 5070 Ti 16 GB (PCIe 5.0 x16), driver 580 → 615.71
• Ryzen 9 9950X3D (16C/32T, AVX-512)
• 64 GB DDR5-4800, EXPO off on purpose (see below)
• ASRock X870 Pro RS WiFi, BIOS 4.10
• Fedora 44, kernel 7.2, Strata in Docker (built from the repo's Dockerfile)
• Model: Qwen3.8-Flash-Next GSQ-RCO IQ2_XS (ISTA-DASLab), Strata v0.1.38 (Oct 9–10) → v0.1.43 (Oct 13+)
• Engine config: 131k context, KV int8, --kv-resident 32768, MTP --spec 4, reasoning_budget_tokens: 8192, conversation cache 4 GiB, vision encoder on CPU
• Also tested locally: Qwen3.8 27B GSQ-RCO IQ3_S on llama.cpp (fits fully in VRAM)

Why the RAM runs at 4800 MT/s. This is my third 9950X3D: the first two died on this ASRock board and went back under RMA. If you missed the saga, the ASRock AM5 + Ryzen 9000X3D failures had several suspected causes:

• ASRock blamed its own PBO/power settings in older BIOS versions.
• AMD pointed at memory compatibility on older BIOS.
• Failures kept being reported after the fix BIOS releases.

Nobody has established a single cause, so I'm not taking chances: EXPO off, stock memory speed, current BIOS. That costs bandwidth. With an MoE model where the CPU computes the experts that don't fit in VRAM, memory bandwidth matters, so the numbers below are probably a few to ~20% lower than the same GPU/CPU with DDR5-6000 EXPO would give. I haven't measured it and won't on this board (propably.. ;) ). If you run a 5070 Ti + 9950X3D with EXPO on, I'd like to see your numbers.

The benchmark

28 tasks, each run twice (56 answers per run). One-shot: no tools, no retries, no agent loop. Every answer is graded by code, with no LLM judge. Code from the answer runs in network-less containers. All data is synthetic (documentation IP ranges, fake account IDs). The mail tasks are in Polish which is worth knowing if you want to compare.

A task passes only when every check passes; the score gives partial credit. The grader has a self-test that grades reference answers (28/28).

Model comparison (Oct 9, same tasks, 2 repeats)

Notes:

• The 2-bit local quant matched the full-precision API model, so I stopped thinking about IQ3_XXS. On 64 GB it would cost RAM headroom and speed for no visible gain on my tasks.
• The 27B is roughly 3–4x slower here and lost overall. It only won at exact counting in logs.
• Flash's weak spots: nginx config details (limit_req inside if), subtle Python (CIDR/IPv6 edge cases), and exact log arithmetic.
• OpenRouter → Alibaba ignored reasoning_effort for Flash. Only reasoning.max_tokens had an effect.

Tuning the local setup, Oct 9–15

All runs below use the local IQ2_XS, reasoning_effort: medium unless stated, and the same 28 tasks × 2.

1. Sampling vs greedy on a 2-bit quant

Several threads here say Strata's greedy default causes doom loops and that you should switch to Qwen's recommended sampling. On my tasks that made things much worse. Medium: 50 → 37/56 (16 tasks lost, 3 gained, sign test p≈0.004).

Most new failures were single-token slips, not worse reasoning:

• A JSON string opened with a Polish „ and closed with ASCII ", which made the JSON invalid.
• Invalid \escape sequences in JSON.
• days instead of noncurrent_days in an S3 lifecycle rule.
• A misplaced paren in jq.

My guess: a 2-bit quant has a flatter, noisier token distribution, so every sampled token has a higher chance of being wrong. With medium reasoning the model has little room to notice and fix it. Lower temperature and min_p reduced the damage but never got back to greedy.

Sampling did fix the loops on high (11 looping answers → 0), but the score stayed at 42 and decode got slower (MTP drafts are accepted less often).

2. "High" reasoning on IQ2 = loops

On high, the model uses whatever budget it gets:

One answer repeated 847 of 903 reasoning paragraphs. Looping text is very predictable, so MTP hits 170–185 tok/s inside the loops, which makes a 16k run finish faster than you'd expect. Even on medium, a 16k budget gave one task back and lost another: the model hand-wrote 128-bit two's complement as 1111…0000… until Strata's repeat detector cut it off with an empty answer. With 8k, the budget wrap-up would have stopped it first. For this quant, the 8k budget works as a safety fuse.

3. Calibration: +25% speed, but read the score carefully

./setup.sh --calibrate (run inside the container, about 25 minutes) measured:

• --pcie-frac: 0 → 124.5, 0.2 → 130.8, 0.35 → 132.9, 0.55 → 131.5, 1.0 → 102 tok/s. It kept my old 0.35. 0.2, suggested in another thread, wasn't better here.
• --spec-min-p: 0.3 / 0.5 / 0.7 → 133.9 / 131.6 / 136.6. It kept 0.7.
• --pool-workers: 15 → 133.5, 10 → 138.0, 8 → 141.6, 4 → 118.2. That's +6%. On a 16-core X3D, fewer workers are faster.
• Adaptive expert tier: default 148.4 → --adapt-every 1 --adapt-swaps 160 --adapt-decay 0.97 159.1 (+7%).

--trim-stage-weights and --pipeline-windows, also suggested here recently, are multi-GPU-only according to the docs, so I skipped them.

On the bench, decode went from a median of 118 to 143–147 tok/s and the full run from 20.6 to 15.4–16.0 min. But the score dropped to 44/56, twice.

Before blaming the flags, I split the failures into two groups:

• hard: the code doesn't work. nginx -t fails, terraform validate fails, hidden tests fail.
• soft: a defensible interpretation or a strict exact-match check.

The hard failures are the same as before (the Oct 13 run with 4 was the lucky one). All of the drop is soft cases:

• An IAM condition of exactly incoming/ instead of incoming/*. The task said "list only these prefixes", so this is a stricter reading of "only".
• Two inbox triage tasks where the model flagged a disk alert as needing action even though a later mail in the same mailbox had closed it.

The calibration changes which experts run on GPU vs CPU and the CPU summation order. The numerical differences are tiny, but greedy decoding flips at near-ties. Those flips then replay identically across runs: the same IAM mistake and the same inbox list in both calibrated runs. So two 44s are closer to one dice roll repeated than to two independent samples. I kept the calibration.

4. Other things worth knowing

• Noise: even with greedy, the two repeats inside one run agree on only ~24 of 28 tasks (Strata's adaptive expert cache isn't bit-reproducible). Treat ±3/56 as noise and look at which tasks flipped, not just the total.
• Vision on CPU ("vision" section in the config, BF16 mmproj, 16 threads) doesn't touch VRAM, so text speed stays the same. A quick check after enabling it gave ~112 tok/s including prefill, in line with the bench. A 1089×787 infrastructure diagram took 17 s end to end.
• Real agent workload (Hermes agent, 54k-token context): about 100–104 tok/s decode, 78–83% expert-cache hit rate, 5–7% of routed experts over PCIe. That was before calibration. (waits for some free time to retest)
• "tool_call_recovery": true in the Strata config is worth turning on for agent use. Per the docs, 98.3% vs 92.9% of tool calls are parsed as intended.
• A Strata config without a sampling block decodes greedy. If your client sends no temperature, that's what you get. Check what your agent actually sends.

Caveats

• 2 repeats per config. One-shot tasks, not agentic. Synthetic data. Some prompts are in Polish.
• Built around my job, so heavy on Terraform/AWS/Cloudflare/mail, light on general coding.
• The cloud-model comparison is a single day. Providers and quants behind OpenRouter can change.
• The findings on sampling and high reasoning are about IQ2_XS. Q4 or IQ3 users may well see the opposite. I'd be curious if someone runs the same comparison on IQ3_S.

Feedback welcome

I'm open to suggestions on what else to test or what to fix in the methodology: other Strata flags, other quants (if they fit in 64 GB), task ideas, grading that's too strict or too lenient. The tasks are synthetic, so I'm happy to share them, along with the grading code, if anyone wants to reproduce this on their hardware or compare against a different setup. Just ask in the comments.

Disclosure: I prepared the benchmark, ran the analysis and wrote this post together with Claude (Anthropic). The runs, hardware and decisions are mine; Claude helped with the bench code, statistics and write-up.

EDITS: Formatting

i had 8 twitter accounts managed through buffer + hootsuite + a va: cost was $340/mo, impressions were ~1200/day and engagement 1.1%. then i got tired of it and decided to just run a local model as the brain of a bot farm.

setup is qwen3.8-27b-instruct-uncensored-v2, q4_k_m gguf (~15gb), running on llama.cpp on my m5 pro (32gb). it reads a yaml config with 37 account personas, picks which ones are due based on their schedule, spawns parallel workers (claude opus via openrouter) to write the actual tweets/comments, then scores every output 1-5 before it goes live.

i chose uncensored specifically cause cloud models always hedging: lecture about GDPR Article 6, CCPA consent and talking to legal counsel. qwen writes like it has opinions and engagement went from 1.1% to 3.8% in two weeks. i think most of that delta is just not sounding like a press release.

the cron line fires every 4 hours:

0 6,10,14,18,22 * * * python3 ~/farm/orchestrator.py --batch >> logs/cron.log 2>&1

a batch takes about 20 minutes. 8-12 accounts get touched per run. each one posts 2-4 tweets across the day (staggered), does 15-25 likes/rt with random 30-90s gaps between actions, comments on 3-5 trending threads in its niche. if one of their own tweets gets >10 replies within an hour the worker goes back and drops 2-3 follow-up comments to keep the thread alive. feels human enough that i haven't been flagged yet.

account creation was the annoying part. 37 signups, phone verification, captchas. wrote a playwright script that spins up headless chromium with a rotating proxy, fills the form from the persona yaml, waits for sms code via webhook. capped at 12/day/ip so it doesn't look like a datacenter burst. three days total. i solved two captchas on my phone while eating cereal. that was the entire human touchpoint for account setup.

the yaml drives everything:

accounts:
  - handle: u/techbro_dan
    persona: "casual tech bro, posts about ai/startups, slightly contrarian, dry"
    schedule: ["09:00", "14:00", "19:00"]
    actions: [tweet, like, rt, comment]
    niche: ["AI", "startups", "Elon"]
  - handle: u/ai_skeptic_42
    persona: "skeptical engineer, questions hype, short sentences, no exclamation marks"
    schedule: ["10:00", "16:00"]
    actions: [tweet, comment]
    niche: ["ML research", "open source"]

37 entries total. each one has its own voice, its own posting times, its own topic pool. qwen reads all of them every batch and decides who's due.

log from a typical run:

[orch] batch 14:00 utc → 8 accounts due, spawning workers
[wkr u/techbro_dan]   post ok (137ch) | like x4 | comment ok | rt x2
[wkr u/ai_skeptic_42] post ok (94ch)  | skip like (no targets) | comment ok
[wkr u/crypto_mom]    rate_limit → backoff 90s → retry ok
[wkr u/devops_kate]   post held (score 2/5: "too generic")
[orch] done. 7 posted, 1 held, 0 failed. next batch: 18:00 utc.

numbers after 14 days vs the old setup:

• accounts: 8 → 37
• tweets/day: ~24 → ~150
• impressions/day: ~1200 → ~14800
• engagement: 1.1% → 3.8%
• my active time: ~45 min/day → ~10 min (check logs, nuke bad tweets)
• cost: $340 → $23 ($19 hetzner vps for ip rotation + ~$4 opus api)

if anyone wants to see the scoring rubric qwen uses or how the persona prompts are structured, happy to paste them here. also curious if other people are running uncensored models as judges/orchestrators rather than generators, feels like an underexplored pattern.

fwiw this local judge / cloud writer split is built into atomic agent as fusion mode: atomicagent.io

I hope this helps someone on the verge of investing for a local LLM.

I forced myself to cancel my Claude subscription to use the investment of Mac Studio here. I haven't been able to sleep with all the things I keep bumping against. A brief summary of what I have setup so far

• vMLX with Qwen3.8 27B 4-bit. Concurrency: 3 sessions, default context window for vmlx which is around 135K tokens.
• LiteLLM as a Gateway to get metrics + an alias with thinking disabled to test.
• Tailscale to reach it from my work computer.
• My workload is mostly infrastructure optimization. Kubernetes management, structured text, logging analysis, MongoDB, BigQuery, a lot of bash interaction.

Qwen3.8 just drifts thinking. It falls into this endless loop of considering stuff. I read posts about this. It really thinks and considers too much. I started to miss Opus here.

Then, the concurrency. I used to have around 10 Claude sessions running investigating stuff. But this is hard with local LLMs, concurrency is really a bottleneck. You either have the models fast or have them concurrent. I learned about speculative drafting and MTP in the first hour. It's all about trade offs. How not everything matches in compatibility is crazy. I tried running speculative drafting qwen flash next but the one has different weights for my setup. I haven't slept in like 30 hours, brain fatigue, trying to justify my purchase. Haha. How did I end up here?

On tooling, this is what I'm using now

• Superset as harness
• OpenCode to reach my Local LLM

This is a cry for help to the community, what do you recommend me doing, exploring, tooling that you have found useful. Interesting open models to test?

Very excited to fully join the Local AI community. Been using Claude Cowork for almost 4 months and run the entire backend of my single employee brewery using PDF extraction etc etc. It changed the game. When Anthropic announced all Scheduled Tasks (which my brewery runs on) were being moved to the cloud versions I had to put my foot down. While I waited for this Ryzen box, I converted my 5070ti to run Qwen 3.6-35b-a3b and it's been pretty good. Not Opus level, but good. Learned Hermes, and what llama.cpp is. Learned why a harness and inference engines are more deeply. It's been a busy week.

Thanks to the communities here and on r/strixhalo I was able to have this sucker running with a Gufo engine, 3.8 Next Flash for that engine, and multiple bots via Hermes running as good as when I started using (Opus 4.6/4.8 times) still more work to do, but I'm almost 100% off the subs, and looking forward to the tinkering!

Been running uncensored models locally for a while now, and I keep going back and forth on this. Wanted to throw my thoughts out and see what you all think.

The case FOR local uncensored AI:

• Zero filter anxiety. I ask a question, I get an actual answer. No "I'm sorry, I can't help with that" on completely reasonable requests
• Privacy is a real thing, not a marketing line. My data literally never leaves my machine
• No rate limits, no subscriptions, no account bans for experimenting with weird prompts
• The models have gotten shockingly good. What needed a datacenter 2 years ago runs on consumer hardware today

The case AGAINST (let's be honest):

• Hardware cost upfront is brutal. Not everyone has 24GB+ VRAM sitting around
• Quantized models still lose something vs full weights vs API models
• You're the sysadmin now. Drivers, backends, model formats, merges — it's a rabbit hole
• Big hosted models still win on raw capability for complex reasoning tasks

My honest take: uncensored local models handle like 80% of what I actually do daily better than the "safe" alternatives, but I still reach for big hosted models when I need heavy lifting. So I end up with a hybrid setup and honestly it feels messy.

Curious about the community:

1. What's your daily driver model right now and why?
2. What hardware are you running it on? (GPU/VRAM/RAM, laptop vs desktop vs server)
3. Did you go local for privacy, censorship issues, cost, or something else?
4. Anyone fully moved off hosted models? What broke?

Just curious where everyone landed on this tradeoff

I was following the LLM space since the days of GPT 3.5 and TheBloke and wanted ever since to have a computer powerful enough to run LLMs locally; about 3 years later I managed to finance a workstation with one RTX PRO 6000 to run Qwen3.8-27B and Qwen3.8-Flash-Next.

And yes, I did name it after the MAGI supercomputer from NGE.

Components used:

• PNY NVIDIA RTX PRO 6000 Blackwell Workstation Edition
• AMD Ryzen 9 9950X
• Gigabyte X870E AERO X3D DARK WOOD
• 2 x Crucial Pro 48 GB DDR5
• Samsung 9100 PRO 4 TB
• 2 x Samsung 990 EVO Plus 2 TB
• 2 x WD Red Plus 6 TB
• Noctua NL-LC1-36
• 4 x Noctua NF-A14x25 G2
• Fractal Design North XL Charcoal Black Tempered Glass
• Corsair RM1000x ATX 3.1 1000W

Already posted this in /macmini - but wanted to get some more localLLM opinions.

I'm seriously considering a Mac Mini M4 Pro 64GB or Mac Studio M5 Max 64GB for local LLMs and automation. But the more I think about it, the less sure I am WHY I actually need one.

ChatGPT, Claude, Copilot, etc. are covering more and more ground: projects, agents, scheduled tasks, file access, automations... And even cloud APIs through Azure or Vertex can offer ZDR.

So what's really left for local AI?

My potential use cases:

• Private documents: Some stuff I simply don't want in the cloud. Models like Qwen or Gemma in the 20–30B range seem good enough for that.
• Knowledge base: Processing documents from past projects, building a searchable knowledge base, maybe using n8n.
• Local automation: Running workflows and agents like Hermes or OpenClaw. But honestly, Power Automate and cloud agents already do quite a lot.

Here's the thing: I'm not a developer. I'm not processing thousands of customer tickets or running some massive automated business. My use cases are mostly personal productivity and business workflows.

And running autonomous agents locally seems to introduce a whole new world of security concerns and maintenance.

So I'm wondering:

What am I missing? What are the actual killer use cases for local LLMs and agents in 2026, especially for non-developers?

Is a dedicated 64GB Mac really worth it? Or would I be better off with a regular 32GB machine, some smaller local models for sensitive stuff, and cloud AI for everything else?

I'm genuinely interested in what people are ACTUALLY doing with their local setups beyond experimenting and tinkering.

Help me out here. Am I overlooking something obvious, or is this just an expensive rabbit hole?

Use cases that I do not see for me are asking the LLM for stories, roleplay or some uncensored stuff or so…

My takeaway so far: The strongest arguments for local AI seem to be privacy, independence from cloud providers, predictable availability, and the ability to run continuous workflows. A hybrid approach using smaller local models alongside frontier cloud models probably makes the most sense for now.

That said, I'm still not seeing any truly compelling use cases that would justify spending several thousand dollars on powerful local AI hardware. Maybe I'm overlooking something or need to rethink how I'm approaching this. I'd genuinely love to hear more concrete examples where a powerful local setup makes a significant difference compared to cloud AI or smaller local models.

Title. Are current small (under 30B) local models on par with old pre 2025 models?

Since thats the conversion rate now. Curious on what the community thinks.

M4 Max 64GB. I had Claude help set me up a workflow and agents.md file to build a small demo for the SNES.

I uploaded tools like the libraries and emulator to a Vendor folder, examples, and told each model to “create a demo for the SNES with a race car driving around a track, top down view)

Claude and ChatGPT did this in a few minutes and generated a .sfc file. I tried it with Qwen and it just got stuck making the graphics after 7 hours. Gemma has already written and tested main.c after 10 minutes.

I expected Qwen to be a little slower but not this slow. Its context was 32K, set to low reasoning.

Have other folks seen this? Is there something I’m doing wrong? Or is it really that slow?

I spent a month making two Tesla P100s fast in llama.cpp — Results : 42% more decode, +74% more on short-prefill.

GP100 is an awkward card for llama.cpp: alone among Pascal chips it has no DP4A, so the quantized matmul paths fall back to emulation and cuBLAS, but it does have full-rate HFMA2 — the one instruction the card is genuinely good at, which llama.cpp barely uses.

The patch set is now rebased onto llama.cpp v0.7.0 (41 patches, mainly for sm_60, MIT, unofficial and unaffiliated with the llama.cpp project).

Measured on 2× Tesla P100-PCIE-16GB, patched vs stock llama.cpp at the same commit (11669 / 620d2225c), 5 interleaved rounds per config with the arm order reversed on even rounds

It is a decode set first: the biggest effect is a reworked internal AllReduce for pre-Volta plus CUDA graphs on sm_60, which is why two cards in a tensor split is where it shines. Prefill is a genuine gain too.

Tested only on 2× Tesla P100-PCIE-16GB, but some patches (mainly prefill and -sm tensor) are generic for all cards, if you can please help benchmark other architectures (Turing-and-later) please share the results.

Apply is a plain patch -p1 -F0 loop against llama.cpp at the matching tag — no scripts, no nix.

• https://github.com/sbendhiba/llama-cpp-pascal-patches

I kept doing the same math every time a model came out, so I turned it into a tool.

Pick your GPU (100 of them, from a PS5 to an H200), a context length and how many requests you serve at once. It tells you:

- whether the model fits, in each format from BF16 down to GGUF Q3_K_M

- which other models fit on your card at the most faithful format that still fits

- what to offload to RAM if it doesn't fit, and how fast it will be with your RAM type

- how many tokens/s to expect, roughly

It handles the architectures where simple calculators go wrong (sliding window, MLA, hybrid linear attention, MoE), and you can paste any Hugging Face model link.

I also added a MCP if you want to use it with your favorite llm !

https://studiotvai.com/llm-vram-calculator

Each model also has a page with its VRAM at every precision and context length, for example
https://studiotvai.com/vram-requirements/qwen3-6-35b-a3b

Free, no account. Feedback welcome, especially if the numbers don't match what you see on your machine.

I've read that many people's experience is that ISTA-DASLab's quantizations are doing better than bigger quant from Unsloth, for instance many people said that IQ3_S from ISTA-DASLab was doing better than IQ4_XS from Unsloth

I'd like to know, for those who tested many quants, how the NVFP4 quant (from Nvidia itself I think?) and Q4_X_L from Unsloth are doing against IQ3_S?

I'm able to run IQ4_XS and NVFP4 at decent speed (>50 t/s decode) but since bigger doesn't necessarily means better (specifically since ISTA-DASLab have their special way of doing it), I'd like to run the best quant I can.

I'm not experienced enough to do benchmarks I truly trust so I'm curious if any of you did some experiments or benchmarks on those!

Seeking advice for the best yield value local setup for coding.

Device: MacBook pro, m5 pro 64GB, 1tb

What: I’m building a compliance based app, using opus5.5 on anthropic subscription to plan, and orchestrate the compliance req, AWS setup, specs etc. with the specs at a high quality I’d like to setup a local LLM to code away, there are roughly 100 modules so it’s a meaty task.

Local LLM skillset: 1/10 I’ve setup Qwen and achieved first prompts - yay! I don’t want to reinvent the wheel on how to make a practical setup to handle the above task, so turning to my online guru buddies to guide the path. I assume that what I’ve setup is suboptimal for the task at hand, hence I’m asking here :-)

Best yield: by this I mean that the local LLM and harness should be relatively quick and easy to setup, I don’t want to spend more time on local LLM setup than I do on the core project. So hoping that there are tried and tested methods to achieve the above.

P.S if you think this idea sucks and I should not use local for this or if I should expand what local does in the workflow, please share, all ears!

🙏

Hi
LMS is updating runtimes frequently, few times a week.
Today they updated em twice!
I wonder, why they dont have a way to simply click a "Purge old runtimes" button to delete old ones?
Current way is to right click on every runtime, select manage versions and manually click on each to delete.
Is there a way to delete old runtimes with one click?

So still pretty new to running local models. I have been running Qwen 3.8 27B and have been pretty impressed with its capabilities. Now I saw the news of this LFM model, and it looks like a faster but less capable model, so I am not sure what the use case of this would be over the other Qwen model.

I don't yet have a frame of reference that I have with the cloud models and by those metrics these are a step down. I guess my question is to know in what situation you use the new LFM model and trying to see more use cases on how people are using local models on their day to days

https://www.liquid.ai/blog/lfm2-5-24b-a2b

I turned a CHIP-8 game console into a decoder-only transformer.

The transformer is the game engine. It executes instructions, updates machine memory, and renders every frame through next-token prediction. Every generated token is a memory write, and your keypresses go directly into the model's context window. There's no separate emulator.

Pong, Tetris, Space Invaders and a bunch of other CHIP-8 games run on the same model. Just swap the ROM in the prompt to switch games.

The weights are compiled from a circuit, not learned through training. Attention handles memory reads by finding the most recent write to an address, while ReLU circuits handle opcode decoding, arithmetic, comparisons and bitwise operations. Even the compatibility quirks are encoded in the initial prompt, so different games don't need different weights.

The model has 18 layers, 42 attention heads and 32M parameters (139k nonzero). It uses fp64 and can be loaded through Hugging Face transformers.

Model weights: https://huggingface.co/divagr1925/chipformer

Try it out: https://divagr.com/chipformer

Code + desktop app: https://github.com/divagr18/chipformer

I'm at that stage now (still relatively new to LLMs) that I have a good number of models downloaded and I've narrowed it down to a couple of models, but what do you use to determine which is the 'best' for coding?

Are there some standard prompts you've got from somewhere or some you've made yourself? I'd be interested to know what you all use.

I'm hoping there is something that can be used to give some sort of benchmark rather than just "getting a feel" of the model.