https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF

https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr-GGUF

NVIDIA Nemotron 3.5 Bolt 74B-A9B 24GB is a vision-language model (VLM) designed to run everyday agentic tasks locally, using a native 2-bit quantization to fit in 24GB of memory and run alongside other applications. It optimizes intelligence and speed per GB. It aims to achieve the optimal balance between speed and intelligence, being faster than most popular dense models while smarter than most popular MoE models with a similar memory footprint.

This model is ready for commercial or non‑commercial use.

Model Summary

Model Overview

Model Developer: NVIDIA Corporation

Model Dates: December 2025 - May 2026

Data Freshness:

• The post-training data has a cutoff date of May 2026.

What is Nemotron?

NVIDIA Nemotron™ is a family of open models with open weights, training data, and recipes, delivering leading efficiency and accuracy for building specialized AI agents.NVIDIA Nemotron 3.5 Bolt 74B-A9B 24GB is a vision-language model (VLM) designed to run everyday agentic tasks locally, using a native 2-bit quantization to fit in 24GB of memory and run alongside other applications. It optimizes intelligence and speed per GB. It aims to achieve the optimal balance between speed and intelligence, being faster than most popular dense models while smarter than most popular MoE models with a similar memory footprint.

This model is ready for commercial or non‑commercial use.

bolt!

From NVIDIA AI on 𝕏: https://x.com/NVIDIAAI/status/2110468004609950057

Yesterday Fish released this incredible text-to-speech model, so naturally I spent the rest of the day building a great demo for them :D The model is a chunky MoE (35B3A), but it runs well on a DGX Spark. The 4bit quants get very close to the original. You can try it here: https://huggingface.co/spaces/fishaudio/Drama-3 or download the model and run it locally 🤗

from Mistral:

Timestral is a family of time series foundation models for zero-shot forecasting, available in 5M, 28M, 123M, and 209M parameter versions. This repository contains the 209M checkpoint.

Timestral supports univariate and multivariate forecasting, past-only and known-future covariates, and probabilistic predictions across 21 quantiles. It accepts up to 32,768 historical time steps and forecasts up to 1,024 steps in a single forward pass.

We’re releasing the source code in our GitHub repository to make it easy to run Timestral and reproduce our benchmark results.

Model Details

• Size: 209M parameters.
• Architecture: patch-based transformer with time attention and variate attention, 24 layers, embedding dimension 768.
• Patching: 32 time steps per patch. Each patch is embedded from its values together with their missing-value indicators.
• Context length: up to 32,768 time steps (1,024 patches).
• Forecast horizon: Up to 1,024 time steps (32 patches) in a single forward pass. Longer horizons are forecast block by block, with each block's median forecast appended to the context.
• Outputs: 21 quantiles (0.01, 0.05, 0.1, 0.15, …, 0.85, 0.9, 0.95, 0.99).
• Inputs: univariate or multivariate target variates, plus past-only and known-future covariates, numeric or categorical (categorical covariates are target-encoded). Missing values are marked as NaN.
• Inference: PyTorch on CPU or GPU.

https://huggingface.co/mistralai/Timestral-209M

https://huggingface.co/mistralai/Timestral-123M

https://huggingface.co/mistralai/Timestral-28M

https://huggingface.co/mistralai/Timestral-5M

this is actually interesting:

https://commandline.microsoft.com/local-models-sandboxed-tools-github-windows/

Microsoft AI developed a local version of MAI Code 1.1 Flash, a coding-optimized mixture-of-experts model with 137 billion total and 6.8 billion active parameters.

Step 5 Preview is StepFun's flagship model for agentic work. It delivers frontier-level performance across software engineering and professional knowledge work, with particular strength in finance.

Will there be a black-friday this year?

day 0 support for two new Nemotrons

minutes later new Nemotron 3.5 Super VL was released:

https://huggingface.co/unsloth/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B

https://huggingface.co/unsloth/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-GGUF

Yesterday I said its hard to play 4 spark with ling-3.1-flash but today its coming!

Hi all,

I've been following this sub for months now, but I recently started working again and missed some of the newer developments with Strata and new model releases since about when Qwen 3.8 27b first came out.

What's the current best set of models / best model to run on a system with lower compute but higher VRAM? I'm currently on M4 Pro with 48gb VRAM and would like to run a local agent on device that still allows me to get basic work done, so something that fits in 24-30gb RAM would be perfect, ideally MoE.

Thank you!

New open model to check out ! Doing great on agentic, focused on arabic.

From 0.9B to 375B !

Weights : https://huggingface.co/collections/IFM/jais-3

Just downloaded the new LFm2.5-24B-A2B LiquidAI model for testing. They claimed some pretty good performance, and I've had good experiences with their prior releases.

Some context - I use Ornith 1.5-35B pretty regularly in multiple workflows - sometimes on it's own, and sometimes as part of an AI-Council in Pi Agent. It's been very reliable, provides surprising insight "smarter" models miss, and is a great workhorse in Pi.

I swapped LFM 24B in to replace Ornith in a few of my workflows - it is holding up REALLY well. It's very fast (~95 tps sustained in llama.cpp using ROCM on an AMD W7900 - latest llama.cpp build). The Q8 takes up 30GB vram with full 128k context. It hasn't failed me yet in my mechanical tasks - it just keeps going and going and going and seems to tick all the boxes for a "little" model I run alongside a larger or smarter model on my other GPU.

For "insight" I'll have to test it longer in my council-mode workflow but I will report back.

Just wanted to give my $0.02 (CAD) on this model - it is very competent with Pi and worth your time to check out. The speed alone is a nice upgrade, and for my more basic tasks, even long running recursive tasks, it is holding up very nicely.

Kudos to the LiquidAI team, this one is going to get alot of use on my chonky boi.

Saw soneone post their bathroom AI PC a few weeks ago. Figured I’d contribute with my chicken-coop adjacent shed AI PC.

Started with a single CMP 170HX (unlocked to 64GB) and ended up with three more for a total of 256gb VRAM. Found a deal on a used Threadripper with a TRX40 motherboard and 256gb DDR4 RAM to host the 4x 170HX. Added a 3050 plugged into riser connected to a NVME slot for video/RDP. My case ran out of space so that’s the board dangling freely lol.

My original goal was to save money by running AI locally. Naturally, this escalated into four 170hx, a 32 core Threadripper, and enough RAM/VRAM to turn my hobby into what my wife says is a financial liability… oof.

Originally had it in a closet, but between the heat and the wife complaining about the noise, the GPUs were eventually banished to the shed. Now they are in a happy place with A/C and 2x 20amp receptacles on separate circuits. One for each power supply. No more fan curves to reduce noise either.

Running QFN split between two of the 170HX with capacitor mods and Qwen3.8-27B on the third. Just got the 4th 170hx installed today. Downloading GLM 5.3 Flash EXL3 4.65 bpw to see how it’ll run on all 4.

Really looking forward to seeing how this beast will handle Qwen4. QFN does so much so well that I can’t imagine what Qwen 4 will be like.

Pulling the benchmarks from the model cards.

Benchmarks might not be exactly comparable, as some models do not specify the version.

I was curious how quantization affects Finnish language ability and why Q38FN’s Q8 quantization got such poor accuracy numbers. So I set the machine running in the background. The Finnish corpus is three books from Project Lönnrot, and the English one is the same books from Project Gutenberg. Wiki is wiki.

The first observation is that Finnish really does take a hit from quantization. If KLD is said to become noticeable after 0.02, then even at Q8 we’re far from that. This measurement doesn’t say anything about how well the model knows Finnish in its bf16 form, only how much worse it gets from there. If a rarer language hasn’t been drilled into the model with large weights, it starts to show when bits are dropped from the precision.

The second observation is that some of the people who quantize models use their own (non-public) corpus to evaluate accuracy. They justify this by saying that wiki has already been trained so well into the n-gram that it would otherwise come through too much. The results are somewhat mixed. KLD on the wiki data is better, but top-1% is worse. In fact, the Q8 on the English data is starting to be what I expected from that level of quantization.

I run a hybrid (“chimera”) model where I’ve swapped Unsloth’s Q4 for a Q8-level n-gram. Someone had measured that it wouldn’t be worthwhile, but by my measurements there’s a clear benefit. The hybrid model lands midway between the Q4 and Q5 models, which is a good result considering I run the n-gram from disk, i.e. it doesn’t load memory at all. A clear observation is also that the size of the n-gram has no effect on Finnish ability.

The charts also show that even a Q5 model would be very close to the Q8 model.

To some extent I miss Gemma-4 31B’s perfect Finnish, but on the other hand Q38FN’s coding skills are in a whole different class.

I continued to use Strata but something just feels off for this an large model, it is very fast but intelligence seemed even under 27B 3.8 UD Q4, it had a hard time coding a basic 3d game and a 2d platformer. Deepseek Flash V4 didn't have this huge and drop off at Q2 imo.

Then I ran across this guys independent testing:

https://lukesdevlab.com/?a=strata%3AISTA-DASLab%3AQwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF.json&b=unsloth%3AQwen3.8-27B-Q4_K_M-GGUF.json

It shows that he needed many shots as well compared to the 27B. But then again Swift 1.5 27b is less smart, in his testing and mine.

Even comparing it to the 27B swift 1.5 we can see it needs more compactions and more shots, for an far larger model.

I would say just use the 27B or regular flash models, avoid Swift or Bottlecap if you are not so time sensitive since original delivers far better results in practice, in my own experience and the tests show it.

Quants do play a part but I've used Q2 and Q3 quants before without this big of an issue, Strata and Swift are sacrificing too much intelligence for speed imo.

TLDR:

Depends on hardware (vram)

Use Next flash strata if you need a very fast coding buddy who needs an exact detailed plan.

Use 27B Q4 if you want the best results and have the time to wait (3x to 5x slower wallclock time)

EDIT: he tested with Q1, I ran at Q3 but still I would say this doesn't even change the final conclusion for me! In actual use it hasn't been an intelligence jump, on the contrary to Q4 27B.

I calculated KLD and perplexity for Qwen3.8 27B models that fit 16GB VRAM:

See comments for the benchmarks. Reddit filters keeps deleting the post.

Conclusion:

q5_1/q5_1 is almost always better than q8_0/q4_0 or q4_0/q8_0.

There is very little difference between q8_0/q4_0 and q4_0/q8_0

q4_0/q8_0 is sometimes better than q8_0/q4_0. So it's safer to just go q5_1/q5_1.

Going up a model quant and running at q4_0/q4_0 is almost always better than running the quant below at f16/f16 in terms of KLD. So basically pick the highest quant you can at q4_0/q4_0. Then if you have room to spare increase KV quant.

Note this applies to lower quants. If you are going from q6->q8 model quant it might be bigger gains increasing KV quant, due to diminishing gains going up model quants. Also this does not apply to other models like Gemma where you have to run f16 KV.

Bonsai and swift quants are included but it does not really make sense to calculate KLD since they are finetunes and the KLD will differ significantly.

I remember great creativity from GPT-2. It was trained on a dataset that is orders smaller than current training datasets.

The problem with overfitting a model to the training data is that it becomes unable to generalise; it is very good to predict the test set, but on fresh data its predictions are wrong. This is what we see with current models, they are extremely good at predicting, especially code, but can't generalise to real creativity.

In theory, a larger dataset should allow to get better predictions on unseen data. This is even more true for neural nets. So my question is: to have a creative model, should a model be trained on a huge dataset, but fitting it less than what is done on current models? Somebody has an idea about training epochs, learning rate, and other parameters of last generation models? Do you think that a model that use a last gen training recipe, but slightly undercooked (or very undercooked) would be better at creative writing and worse at coding?

This model is a 74B parameter, 9B-active-parameter mixture-of-experts (MoE) model, packed into 24GB using 2-bit quantization with advanced quantization-aware training techniques to retain quality. It aims to achieve the optimal balance between speed and intelligence, being faster than most popular dense models while smarter than most popular MoE models with a similar memory footprint.

Nemotron 3.5 Bolt E4B 4GB releases Oct. 23

Additional detail

Discovered by a third party… Tencent itself doesn’t seem to be talking about it, which is slightly concerning.

I saw the 4x R9700 build for a 10-person startup here a few days ago and wanted to share my results, since I opted to spend most of the money on the cards. I went with a super cheap used EPYC instead of a Threadripper. In the end it came to about $10.7k per box all-in, and I built three of them for a total of 384 GB of VRAM.

As a long time lurker on my other account, I'm always frustrated that people never show how they actually use the models, so after a lot of back and forth I decided to share the actual project they power instead of being vague.

I use these to run the AI behind my business, USA Arms. We organize tens of thousands of firearms-related businesses (gun dealers, ammo dealers, ranges, etc.) and that info is always changing, so we need a system that keeps the site up to date. The models check source facts, sort out business identities, and pull out stuff like services and hours. Every answer gets verified against what the agent actually saw, and we store the documentation so bigger models can audit the work.

I didn't plan to move the majority of my work to local AI until later this year, but hosted models started refusing to do normal research for my business ~2-3 weeks ago. They refused to identify a gun dealer because the license holder might be an individual, and wouldn't look at an ammo dealer's website because they can't help anyone buy ammo. Our harness just started breaking in random ways, so I'm glad I made the leap.

We tested Luna 6, DeepSeek, GLM 5.3, Qwen 3.8 27B, Gemma and Inkling on our workload, along with Kimi, GLM Flash and others. GLM gave us the best results. Qwen gave us the best cost-to-performance, so that's what these run.

Enough ado, into the numbers:

Per box, $10,733.91 with tax and shipping (averaged over the three)

• 4x ASRock Radeon AI PRO R9700 Creator 32 GB (128 GB VRAM), each capped at 210 W: $7,244.58
• AMD EPYC 7452, 32 cores, used: $127.17
• ASRock Rack ROMED8-2T/BCM: $1,162.39 (there was a global shortage, ours came from Helsinki)
• 8x 16 GB DDR4-2666 ECC RDIMM, all eight channels: $860.59
• SilverStone HELA 2050R Platinum: $578.92
• SilverStone RM44 4U: $388.40
• Samsung 990 PRO 1 TB: $254.39
• ARCTIC Freezer 4U-M, 3x ARCTIC P12 Pro front, 2x Noctua NF-A8 rear, SP3 frame, paste: $117.48

The GPUs are 2/3 of the cost, but the boxes also run our CI/CD, which had become a bottleneck in my businesses. I hadn't seen anyone discuss this: they run Rust builds just fine if you tune them to prioritize inference. In our test, running builds alongside inference didn't measurably change throughput, and the builds were only 3-8% slower.

Engine

• vLLM-Radiance 1.0.387 (vLLM 0.30.0, ROCm 7.14). Huge thanks to the Radiance folks; their R9700 work is what we run.
• Qwen/Qwen3.8-27B-FP8 (Qwen's official FP8 checkpoint), FP8 KV cache, the model's own MTP head with 8 draft tokens
• Two engines per box, TP=2 each, 8 sequences per engine, 256K max context, about 488K tokens of KV cache per engine

Results, vllm bench serve, one box, 2x TP2:

• 1 user, MT-Bench: 53.1 tok/s, TTFT p50 125 ms
• 16 users, MT-Bench: 484 tok/s aggregate, about 40 tok/s per user
• 16 users, random 1024 in / 512 out: 623 tok/s aggregate, 837 W average for the four cards
• 32 users (a bench profile with max-num-seqs 32): 603 tok/s

MTP depends a lot on the text. On our JSON extraction, 83% of drafts get accepted (7.6 tokens a step), vs 39% on MT-Bench (2.7 a step), so MT-Bench undersells it for structured output.

Long prompts, 8 unique real-text prompts at once (4 per engine), 128 out:

Every prompt was unique (0% prefix cache hits).

FP8 vs MXFP4 on a real workload

The startup build ran MXFP4, so I had AI run that too (Radiance's fastest profile) on one box against our own graded extraction workbench:

• FP8: 98.0/100 correct, 4,477 validated jobs an hour
• MXFP4: 95.3/100, 4,819 an hour
• On a holdout set: FP8 94.0, MXFP4 90.0

So FP8 roughly halves the wrong answers for 2–7% less validated throughput. Sticking with FP8 was an easy call for this workload, where "false facts" that make it into the system are a real headache.

For reference, our six Mac Studios (four M4 Max 128 GB, two M3 Ultra 512 GB) on MLX 8-bit scored 99/100, but at 723 an hour across all six. Plan A was buying more Macs, and keeping up would have taken at least 41 more, about $280k.

I made a video of the build targeted more at beginners and people trying to move AI locally: https://youtu.be/o7k5Kj-ho-M

Happy to answer questions. The first of which I expect to be how the heck were the Macs so slow, and I'm sure I'll get flamed haha.

Mixing Nvidia+AMD hardware via RPC usually meant split-mode layer only.

I managed to keep split-mode tensor working, and patch my llama-rpc to send the drafters over to my strix.

That let me go from 150k -> 195k context, at the same tensor parallel speed.. nice.

My agents helped with a TON of this setup:

- The 10Gbps usb NIC just came out in sept. The drivers are non existent, but a gpt session later we were git pulling and modifying 2 files to build it for my correct OS version.

- The llamacpp side took merging 2 PRs + 1 local patch. Its not even worth submitting a PR for the 3rd patch since PRs go dormant now, so Ill leave it here so your agent can build you a patch at whatever state the main branch is in.

In theory this could be offloaded to anything with 2-3gb

[35051] 0.06.133.020 I load_tensors: offloading 4 repeating layers to GPU
[35051] 0.06.133.020 I load_tensors: offloaded 6/6 layers to GPU
[35051] 0.06.133.024 I load_tensors: RPC0[10.44.0.6:13315] model buffer size =  1821.88 MiB
[35051] 0.10.019.793 I llama_context: constructing llama_context
[35051] 0.10.019.798 I llama_context: n_seq_max             = 1
[35051] 0.10.019.799 I llama_context: n_ctx                 = 190208

# ============================================================================
# Qwen3.8 DFlash2 tensor target + remote RPC drafter
#
# Known-good llama.cpp base:
#   77c4540ce4667f4577753fe034a3a8042f510b8f
#
# Changes:
#   1. PR #26636 commit 1a6fee5
#      Allows draft context to use backends required by shared target tensors.
#
#   2. PR #27858 port commit 1784fc3
#      DFlash2 tensor-parallel CPU selector support.
#
#   3. Local patch
#      Always use DFlash2 CPU selector, allowing target=CUDA0,CUDA1 tensor
#      while draft is pinned separately to RPC0/coal.
# ============================================================================

BASE=77c4540ce4667f4577753fe034a3a8042f510b8f

git switch -C exp-dflash-tensor-rpc "$BASE"

# PR #26636
git fetch origin refs/pull/26636/head:pr-26636
git cherry-pick 1a6fee5

# PR #27858 newer port
git fetch https://github.com/henrique-simoes/llama.cpp.git \
    mv/dflash-tp:henrique-dflash-tp
git cherry-pick 1784fc3

# Local RPC DFlash selector patch
cat > /tmp/dflash-coal-rpc.patch <<'PATCH'
diff --git a/common/speculative.cpp b/common/speculative.cpp
index 775df85c7..e76caef0b 100644
--- a/common/speculative.cpp
+++ b/common/speculative.cpp
@@ -1062,7 +1062,7 @@ struct common_speculative_impl_draft_dflash : public common_speculative_impl {
         selector_top_k = llama_model_dflash_selector_top_k(model_dft);
         is_dflash2     = selector_top_k > 0;
         // under tensor parallelism the lm_head is split and the in-graph selector cannot run
-        is_dflash2_cpu = is_dflash2 && llama_model_get_split_mode(model_dft) == LLAMA_SPLIT_MODE_TENSOR;
+        is_dflash2_cpu = is_dflash2;
         mask_token_id = llama_vocab_mask(llama_model_get_vocab(model_dft));

         if (is_dflash2_cpu) {
diff --git a/src/models/dflash.cpp b/src/models/dflash.cpp
index e707c6b03..7942058fe 100644
--- a/src/models/dflash.cpp
+++ b/src/models/dflash.cpp
@@ -153,7 +153,7 @@ void llama_model_dflash::load_arch_tensors(llama_model_loader &) {
         // Under tensor parallelism the lm_head is split along the vocabulary, which the
         // in-graph selector cannot consume, so DFlash2 ranks on the CPU and reads these tables
         // straight from the GGUF file; otherwise the in-graph selector uses them in the model
-        const bool use_cpu_selector = params.split_mode == LLAMA_SPLIT_MODE_TENSOR;
+        const bool use_cpu_selector = true;
         dflash_selector_prev   = create_tensor(tn(LLM_TENSOR_DFLASH_SELECTOR_PREV,   "weight"), { rank, n_vocab }, use_cpu_selector ? TENSOR_SKIP : 0);
         dflash_selector_next   = create_tensor(tn(LLM_TENSOR_DFLASH_SELECTOR_NEXT,   "weight"), { rank, n_vocab }, use_cpu_selector ? TENSOR_SKIP : 0);
         dflash_selector_hidden = create_tensor(tn(LLM_TENSOR_DFLASH_SELECTOR_HIDDEN, "weight"), { n_embd, rank }, 0);
@@ -807,7 +807,7 @@ llama_model_dflash::graph<false>::graph(const llama_model & model, const llm_gra
     // DFlash2 under tensor parallelism ranks on the CPU from the gathered lm_head output, so
     // the decoder hidden states ride the nextn slot; without tensor parallelism the in-graph
     // selector packs its lattice there instead (see below)
-    if (model.dflash_selector_hidden && model.split_mode() == LLAMA_SPLIT_MODE_TENSOR) {
+    if (model.dflash_selector_hidden) {
         res->t_h_nextn = cur;
     }

@@ -862,7 +862,7 @@ llama_model_dflash::graph<false>::graph(const llama_model & model, const llm_gra
     }

     // DFlash2 ranks its candidates in-graph when the lm_head is not split along the vocabulary
-    if (model.dflash_selector_hidden && model.split_mode() != LLAMA_SPLIT_MODE_TENSOR) {
+    if (false && model.dflash_selector_hidden) {
         build_dflash2_selector(*this, model, inp_tokens);
     }
 }
PATCH

git apply /tmp/dflash-coal-rpc.patch