I've used this to watch shows and movies that have never been translated to English. There's so much great content out there, that is now accessible thanks to AI.

Basically you clone the repo and tell your local model (or non-local model) to direct the dubbing process; the rest of the tools are all local as well (OmniVoice, IndexTTS, Chatterbox, Whisper). The repo contains all the scripts and guidance for the model to do a successful dub in the actors' own voices. It's also instructed to improve its own scripts so feel free to push back improvements so everyone benefits. Thanks.

Anime sample: https://www.youtube.com/watch?v=qXosYS-cTXk

Movie sample: https://www.youtube.com/watch?v=2DsFAoTbQ8E

Repo: https://github.com/swax/dub-anything

Hey everyone, I’m trying to decide what to do with my local LLM setup before putting more money into it.

A bit of background: I’m a final-year electrical engineering student, and I use local models for learning about LLMs, experimenting stuff and my thesis on financial analysis of solar plants with BESS. I also am going to work in strategic consulting after graduation, so I want a setup that is useful for data and market analysis and some coding.

My current machine:

• i7-7800X
• MSI X299 Raider
• 128GB RAM — 8×16GB DDR4 2666MHz
• 2× RTX 3090

I’m currently running Strata IQ3_S at around 100 tokens/s, and I’m pretty happy with the speed. Quality seems okay, a bit better than 27B at Q8.

I also have five more 3090s: one ASUS ROG Strix and four Zotacs. All five need repasting and new thermal pads. So seven cards total, with only two currently in use.

I paid about $900 each for the four Zotacs. That’s part of why I’m torn: I could use them for a bigger setup, but I could also sell them for around $1,300 each pretty easily in my local used market.

These are the options I’m considering:

1. Upgrade the existing X299 machine

I found an i9-10940X on AliExpress for around $300. It would give me more cores for my other server workloads and more PCIe lanes, without replacing the motherboard and RAM.

My understanding is that I could then run the GPUs at x16/x16, depending on the motherboard’s slot wiring. Please correct me if there’s a limitation I’m overlooking.

Alternatively, I could use more widely spaced slots at x16/x8. I’ve tested that layout, and the top GPU runs about 10°C cooler, with the cards sitting around 65°C and 58°C.

Would you spend $300 on this CPU upgrade, or avoid putting more money into X299? For inference, would you prioritize x16/x16 or the better cooling at x16/x8?

2. Build a separate machine for the four Zotacs

This would give me a second machine with 96GB of VRAM. I’d need to buy a suitable CPU/motherboard setup and sort out cooling, power, and the rest of the build.

For anyone who has gone from 48GB to 96GB:

• Was the improvement in model quality worth it? Im looking at maya for GLM 5.3 flash.
• Which models and quantizations made the upgrade worthwhile?
• What inference speeds are you getting, and with which backend?
• What reasonably priced platform would you recommend for four 3090s? Used hardware is fine.

I’m happy with my current speed. The main reason to expand would be access to better models or higher-quality quantizations for engineering, financial modeling, document analysis, and coding.

3. Sell the Zotacs and keep the current setup

Since I paid about $900 per card and could sell them for around $1,300 each, I could also sell them and avoid the cost and work of another build.

Would you keep the cards because 96GB opens up enough useful possibilities, or sell them while the local resale price is still decent?

It runs at around m4 16 at 60 tps at full 130k context, you can easily run a Q8 if you have a 16 GB GPU, I was able to get 140TPS on 9070xt,

It’s an MoE so I think we can run it well on smaller cards too, 8GB, 12GB should be possible to get upwards of 30-40 tps.

I have used it a bit, it can code, but I think having a bigger model plan tasks in detail would make it quite good for quick task execution.

I'm planning on maxxing out my pcie slots with rtx 6000 blackwell cards for tp4. I'm wondering if its possible to use an egpu dock like the razer v2 and connect it via thunderbolt to usb 4 on the back of my mobo. The egpu in question would be another blackwell card (5070ti), and its sole purpose is for sidecar models, and to act as a display card for my rig so that i dont have to go headless.

Do you guys know if this is possible?

I read this thread recently and it kind of scared me off:
https://www.reddit.com/r/LocalLLaMA/comments/1vlfe9f/egpu_folks_rtx_pro_6000_blackwell_egpu_crashes/

Can anyone think of a good use for this if it ends up being true? Unless the workload can somehow be split between the NPU and a dedicated GPU, I suspect it'll only be able to run small models.

Sampling: temp 1.0, top_p 0.95, top_k 20 for both. Run with evalscope over the OpenAI-compatible API: Qwen3.8-Next on Strata (MI50 32GB), Qwen3.8-27B on llama.cpp (RX 7900 XTX 24GB).

Caveat: These are only a portion of the full sets, since they take days.

Edit1: Strata started with --kv fp16 and llama's default f16.

Results

Sign test on the 14 problems only one model solved (10 vs 4): two-sided p = 0.18, so a lean toward Next rather than proof.

Token limits

Failures: wrong answer vs out of tokens

BFCL v4 by category (7 per category)

Speed (AIME + LiveCodeBench generation)

Open-weights ASR + full-duplex turn control in one model. It hears when you finish, when you interrupt, and when you're just saying "mm-hm".

• 250 ms median endpoint latency, 80 ms frames, no lookahead • Full voice agent tyger-s2s: TYGER + Gemma 4 E4B + VoxCPM2, one 32 GB GPU, up to 4 conversations • English only for now

🤗 https://huggingface.co/nineninesix/tyger 💻 https://github.com/nineninesix-ai/tyger-s2s

GPU- i have asymetric architectures - Titan RTX(a 2018 card released same time as 2080ti) and i have a RTX 3090. Both of these cards are 24gb the TitanRTX does not have flashAttention-2 only standard flash attention

SystemRam - i have about 48gb of Free SystemRam to spare. the rest is used up by my chrome VsCode and docker instances. 6000mhz ramspeed.

Im getting 32tok/s on qwen3.8-27b running on a single 3090 at 300k context the Titan Rtx is pretty much empty and just sitting there.

but im wondering is strata designed for multigpu will it massively decrease my prefill speed on multigpu? Anyone tested?

is it faster on a single 3090 and the rest in system ram or would it be faster on dual cards even with my specific caveats?

I spent some time taking best practices from the past two years developing my projects with agent skills and centralized them in a repo so I can reuse them and standardize my projects. In the process of doing that, I realized that skills exist in different scopes and timelines, so I spent some time trying to figure out how to organize skills usefully. I realized that Magic The Gathering abilities have a lot of parallels to agent skills, so I did some experiments separating skills into three categories that MTG uses: activated, static, and triggered. I found this works very well for communicating to agents considerable context about the skill scope with minimal verbosity.

If you're not familiar with MTG abilities, firstly MTG is a complex card game, and secondly each card in the game has some text at the bottom which amounts to a special coding language. The coding language's main block is called an ability, which represents some effect in the game. Abilities are separated into several types which set the scope and timing of the effect. These ability types are directly transferable to skills:

• Activated Skills: These are skills the agent must choose to use, and are a set of procedures that should be performed in order once
• Static Skills (called "passive" in my repo): These are skills which set general rules that should be applied all the time given the agent is working in a particular domain.
• Triggered Skills: These are skills which are like activated skills, but they should always happen any time a skill-defined event occurs. The event can be anything: something the agent does or an external event like a time passing.

Separating the skill into these categories lets the agent have some preconceived ideas about how the skill should be used, which increases reliability and decreases context redundancy.

I thought this was an interesting notion and just wanted to share. If you check out my skill repo, it has some other good ideas I've collected over time like a hermes-inspired skill self improvement system, test process organization, and documentation style/methodology. If you have any ideas about making it better, let me know.

I have a 3rd gen Threadripper paired with this board, which annoyingly has only three PCIe slots.

https://rog.asus.com/motherboards/rog-strix/rog-strix-trx40-e-gaming-model/

I am currently running 3x 3090's at PCIe 4.0 x16 each and I get around 100 t/s using QFN on Strata. Now I wish to add a 4th 3090 to run other models and I am not sure whether to use bifurcation or upgrade the motherboard, or the entire system (mobo+cpu).

I had a look at the c-payne SlimSAS cards to bifurcate to x8/x8 but once shipping costs are considered these tend to work out to be rather expensive. There are some Chinese counterparts on Amazon, Alibaba, etc. but I am unsure about how reliable these are.

The other concern I have is whether using x8 will lower performance on Strata since it seems to be more reliant on the system as a whole and not just the gpu's. Would appreciate some advice on how to add the 4th card.

We just shipped web search in Private Mind, our open source app for running LLMs locally on iOS and Android. The video is the whole pitch: Qwen 3 1.7B with web off makes up Apple's September event, with web on it gets it right and cites 5 pages.

The fun constraint is that a 1.7B model with a short context can't just read 5 web pages. So most of the work is picking the few hundred words it actually gets to see.

Roughly how it works:

• There's no search API. A hidden WebView on the phone opens DuckDuckGo (Brave and Mojeek as fallbacks) and we parse the results page
• It opens the top pages and cuts them into ~500 character chunks
• An on-device embedding model (LFM2.5, ~415MB) plus some keyword matching picks the best 6 chunks, max 3 from one page
• If only one site claims something, it counts for less than something two sites agree on
• The model answers from those chunks and links every source

Web search is a per-chat toggle and off by default. When it's on, the phone makes the same requests a browser would. The chat itself stays on the device.

Everything runs on react-native-executorch, both the LLM and the embeddings.

Repo: https://github.com/software-mansion-labs/private-mind

This is an offline app with 51 hands-on labs for learning how AI actually works, from neurons to agents and more...

**A bit of insight**

- 51 labs in 6 groups, from the basics (what a neuron is, gradient descent) through attention, RAG, agents, fine-tuning, quantization, serving and more

- Each lab has a short lesson beside it, readable in Plain or Standard mode

- Some of it actually runs rather than just animating:

- the attention lab runs a small trained transformer (1.37M params) inside the app

- the training labs train a tiny character-level model live as you move the sliders

- Works offline: no account, no telemetry

**How it was made**

I chose the topics, some background research and came up with the structure and the grouping. Claude and GPT Astra to help write the lesson text, and Grok as a second pass on references. There will be mistakes, so if you spot one, please tell me or open a PR.

If you teach this or work in a specialized area of AI, I'd love help expanding it: a new interactive lab, a better visualization, or a tweak that makes the cause and effect in an existing lab clearer. I'd also like to hear which labs are confusing and what's missing.

GitHub: https://github.com/Fazmin/AILearningGuide

I've been working on an inference engine called jitllm. The idea came from my background in operating systems and JIT compilers: generate kernels for whatever hardware is there, and treat layers a bit like processes the engine can place and move around.

In the video, a cloud VM with T4s: mid-reply, all layers of Llama-3.2-3B move from one GPU to the other and back. And my puny laptop where the model is split across CPU, CUDA and Vulkan (iGPU), rearranged while it's generating. The KV cache is paged and moves with the layers, so the prompt isn't re-processed and the output matches a run that didn't move.

What excites me is the room the design leaves. Generated kernels mean every improvement reaches every machine, and the engine can fuse and tune for hardware it has never seen. Paging and relocation turn placement into scheduling: models bigger than memory, mismatched GPUs, work that shifts as conversations grow.

Code -> https://github.com/jitllm/jitllm

Inspired by how short-term memory works in our own brain, I built PulsarLM, an inference engine that removes old and unimportant tokens from the context window to keep it bounded while also allowing the conversation to go way past the trained limits.

Running Qwen3 8b on a 5090, I get 6.7k tok/s during prefill and 89 tok/s during decode. It does support batched decode and with 16 concurrent requests, I get 1.2k tok/s.

It works by using custom attention kernels that accumulates an attention mass per token page. At regular intervals, pages that receive low attention mass gets removed.

To workaround out-of-distribution positions, embeddings are re-rotated (for RoPE) such that token positions stay contiguous and never exceed the context limit.

Eviction of tokens in the context window would also break prefix caches normally, but PulsarLM also caches snapshots of context to avoid unnecessary re-prefills, but does use up extra RAM. There is also an option to persist those snapshots.

Only a smaller and older models are implemented for now, and the engine itself still has rough edges. It's only been tested on linux x86_64. Feedbacks are welcome! Also, what models would you be interested to try?

Here's the link: https://github.com/cosmonomic/pulsarlm

Edit: There used to be a recall mechanism using bm25+attention score, but removed it since it didn't make sense to store that many KVs. So right now, it works best when paired with an external memory system/RAG.

And also, running benchmarks right now, will update once results are out.

For anyone interested, I've been doing some kernel and tuning optimization for AMD because it was performing abysmal out of the box. But the effort was worth it as I was able to more than 2x its performance out of the box (the card is extremely new and there are almost no GEMM table and supports yet). (due to both using separate models and environmental factors input token can have ~3% less tokens for AMD to decode despite effort to keep them as close as possible). Just to keep things short and concise, I did a varying tests of small input c=8, medium input c=20, and long input c=1. Everything was done in cold state (no kv-cache at start).

And I think there is more performance to unlock for MI350P as below benchmark was performed when I was only able to unlock 50% mem utilization (currently inference is still heavily bounded by compute because of missing GEMM tuning config)

Model used:
AMD: amd/Qwen3.8-27B-Quark-AWQ-MXFP4
Nvidia: unsloth/Qwen3.8-27B-NVFP4

(Small) 3k-input, 8 user comparison, 100 prompts
AMD:
Total input tokens: 288653
Total generated tokens: 75000
Nvidia:
Total input tokens: 300000
Total generated tokens: 75000

(Medium) 50k-input, 20 user comparison, 20 prompts

AMD:
Total input tokens: 965153
Total generated tokens: 10000
Nvidia:
Total input tokens: 1000000
Total generated tokens: 10000

(Large) 200k-input, 1 user comparison, 5 prompts
AMD
Total input tokens: 962033
Total generated tokens: 25000
NVIDIA
Total input tokens: 2000000
Total generated tokens: 25000

[UPDATE / RETRACTION: The "100% GTO" claim had a prompt-leakage bug] We patched the repo to pass strictly neutral states, here are the real, honest numbers: • Latency: Still ~2ms (ONNX + Absorbing Markov Chain). • True Zero-Shot GTO Parity: ~15% - 29% across all 340 canonical hands.

**[Benefits of Tier 0 Absorbing Markov Chain + Embedding model] on JevBench, Tier 0 achieved 12 / 12 early exits with 100% accuracy at 16.5 ms p50, and on Decision Index 0.3, it safely rejected 344 / 344 out-of-domain items with 0 false exits

Gevva0 Cascading Decision Engine:

Tier -1 · Static Rules (<50 µs): Hot-reloadable regex and keyword guards that run in microseconds without touching the GPU

Tier 0 · Absorbing Markov Chains (<3 ms): Embeds queries into a 768D manifold using EmbeddingGemma-2 (740M) and solves random-walk absorption probabilities in closed form, hitting 100% accuracy on early exits with zero false exits on 344 out-of-domain items.

Tier 1 · Fast-Path Logits (~35 ms): Runs a single forward pass on Gemma 4 26B-A4B (MoE, 4B active) with a custom K=64 micro-head that slices the 262,144-row vocabulary down to just the 52 answer letters cutting output-head compute by 99.98% and resolving 77.9% of queries without generating a single token.

Tier 2 · Hot-KV Bounded CoT (≤ 256 tokens): Escalates the hardest 17.1% of requests into bounded Chain-of-Thought reasoning directly from the KV cache Tier 1 already prefilled (reuse_kv=True), eliminating double prompt processing and cutting long-context latency by up to 30%.

Highlights:

• JevBench, Gevva0 beats Claude Opus 4.6 Thinking on chance-corrected intelligence (84.62 vs. 83.90) at a 44 ms median latency (vs. Opus's 4,850 ms)
• Decision Index 0.3, our 4B-active MoE cascade outperforms dense 27B models (like Kev 27B) at a 60 ms p50—roughly 7× faster than 27B alternatives.
• 60 FPS Real-Time Game-Loop Ready (Pacman demo included)

https://github.com/solvingSteve/Gevva0

a new option to optimize your VRAM

Are we seeing early signs of the bubble popping?

I'm using Qwen 3.8 27B on an R9700 and it works great, occasionally I'll switch to Tiel-coder for rapid simple tasks. One big problem I run into that makes real projects come to a halt is that more code takes up more context, compaction becomes more and more frequent between any edits, until it gets stuck in a compaction loop not doing any useful work. I started using Q4_K_XL and Q8 kv cache, for a context of 200k tokens, which helps from my normal 100-150k, but is there a technique to ensure it works efficiently?

One idea I have but never fully enforced is having the code indexed (function signatures with comments and line numbers, structs shown in full so it doesn't need a read, maybe methods listed under structs.) Or another way is to make the code extremely modular so the LLM will have a very narrow focus. I tried this but it becomes very cumbersome because other modules will also need changed, probably until each becomes mature enough?

I'm sure some here read this at the time and remember, but I recently found this hilarious Guardian article from 2019 through Wikipedia:

https://www.theguardian.com/technology/2019/feb/14/elon-musk-backed-ai-writes-convincing-news-fiction

Some choice quotes:

"OpenAI ... says its new AI model, called GPT2 is so good and the risk of malicious use so high that it is breaking from its normal practice of releasing the full research to the public in order to allow more time to discuss the ramifications of the technological breakthrough."

"One is its size, says Dario Amodei, OpenAI’s research director. ... It was trained on a dataset containing about 10m articles, selected by trawling the social news site Reddit for links with more than three votes. The vast collection of text weighed in at 40 GB, enough to store about 35,000 copies of Moby Dick."

Some people never change!

Gguf's: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B-GGUF

Free Demo: https://playground.liquid.ai

"LFM2.5-24B-A2B is a 24B-parameter reasoning model with only 2B active parameters, built for fast, private agents on-device. It is the largest and final model in the LFM2.5 family, which now spans two orders of magnitude: from LFM2.5-230M to LFM2.5-24B-A2B.

• Best-in-class agent: Competitive with Qwen3.6-35B-A3B, a model with ~50% more total and active parameters, on agentic and tool-use tasks, scoring 48.09 on LiveClaw versus 38.19.
• Reasoning at MoE speed: Always thinks before it answers. With only 2B active parameters per token, each reasoning token is cheap.
• Efficient inference: Runs up to 1.7× faster than comparable MoEs on-device, with 184 tok/s on an Apple M5 Max and up to 21K output tok/s on a single NVIDIA B300 GPU.

LFM2.5-24B-A2B is a general-purpose text-only reasoning model with the following features:

• Total parameters: 24.0B
• Active parameters: 2.45B
• Number of layers: 40 (30 double-gated short convolution blocks + 10 GQA)
• Experts: 64 per MoE block, top-4 routing
• Training budget: 36 trillion tokens
• Vocabulary size: 128,000
• Context length: 131,072 tokens
• Languages: English, Arabic, Chinese, French, German, Italian, Japanese, Korean, Portuguese, Spanish, Vietnamese, Thai, Indonesian, Hindi, Russian, Polish
• Generation parameters:
  • temperature: 0.1
  • top_k: 50
  • repetition_penalty: 1.05"