Nemotron 3.5 Bolt 74B-A9B VRAM requirements: does the "24GB" model fit a 24 GB GPU?

Nemotron 3.5 Bolt 74B-A9B is NVIDIA's local agent model: 74B parameters in total, 9B active per token, a Mamba-2 and Transformer hybrid mixture-of-experts with image and video input, compressed from Nemotron 3.5 Super VL 120B-A12B.^[2] NVIDIA released it on Hugging Face on October 15, 2026 under the OpenMDW-1.1 license, with a context of up to 256K tokens.^[2] Hugging Face counts 74,885,081,096 parameters in the BF16 checkpoint.^[1] It is popular for a model this new: about 201,713 downloads in the last 30 days for NVIDIA's 24GB GGUF repo, plus 69,763 for Unsloth's GGUFs (checked November 3, 2026).^[4][3]

Why the "24GB" model needs more than 24 GB

NVIDIA built this model for a memory budget. Its card says the architecture search "targets a deployment footprint below 24 GB at batch size 1 and 128K context", and that the routed and shared experts are stored at about 2 bits (IQ2_XXS, 2.0625 bits per weight), with Mamba layers in NVFP4 and the embedding and output layers in Q8_0.^[2] The file that came out of that is 22,018,326,528 bytes (22.0 GB, or 20.5 GiB).^[4]

But NVIDIA's own run commands load more than that one file:^[2]

File sizes from Hugging Face.^[4][14] Then the engine adds the KV cache, the Mamba-2 state and its working buffers. NVIDIA's deployment table gives two budgets:^[2]

Neither recipe is listed for a 24 GB card. Our sums below come in under NVIDIA's budgets because we don't count the engine's working buffers, which grow with the batch settings, or the drafter's own cache; NVIDIA doesn't itemize those. You can try a 24 GB card with the model file alone (no --mmproj, no drafter) and a short context, but our numbers put it above 90% of the card, so expect to cut context or offload a little to system RAM.

Memory needed at each quantization

Below is NVIDIA's file, the NVIDIA recipe total, and every build in Unsloth's GGUF repo, using the exact file sizes on Hugging Face (1 GB = 1,000,000,000 bytes, multi-part files added together), largest first.^[4][3] Unsloth's plain IQ2_XXS and UD-IQ2_XXS files are within 71 KB of NVIDIA's 22.0 GB file, so they are not listed separately. There is no Q4_K_M build; the 4-bit option is UD-Q4_K_XL at 50.2 GB.

Which file should you pick? NVIDIA trained this model to be run at 2 bits (quantization-aware distillation), so its own 22.0 GB file is the intended one, not a last resort as 2-bit usually is.^[2] The larger Unsloth builds are quantized from the BF16 checkpoint and need 28.8 GB (UD-Q2_K_XL) to 78.9 GB (Q8_0). We have not seen a published quality comparison between them and NVIDIA's file.

Context length adds memory (KV cache and Mamba-2 state)

Only 8 of the 88 layers are attention layers. From config.json (2 KV heads, head size 128), an FP16 cache takes 2 × 8 × 2 × 128 × 2 bytes = 8,192 bytes per token: about 0.07 GB at 8K, 0.54 GB at 64K and 2.15 GB at the full 262,144 tokens.^[15] NVIDIA's 28 GB recipe stores it in 8-bit (q8_0), which roughly halves that.^[2]

The 40 Mamba-2 layers keep a fixed-size state instead of a growing cache. This part is our estimate: with llama.cpp's sizing (convolution state (4 − 1) × (8,192 + 2 × 8 × 128) and SSM state 128 × 8,192, both stored as 32-bit floats) it comes to 40 × 4,317,184 bytes = about 0.17 GB per sequence, whatever the context.^[15][18] We assume NVIDIA's GGUF keeps the BF16 checkpoint's layer layout. The table adds both.

That is why the context barely moves the total: going from 8K to 256K adds about 2.1 GB, less than the image projector and drafter files together.

Which GPUs and Macs it fits on

Pick your GPU or Mac and a context length to see which builds fit. The weights plus the KV cache and Mamba-2 state are compared with the device's memory.^{[5][6][7][8][9][10][11][12]}

How to read it: "Fits" means the total is under 90% of memory. "Tight" means 90–100%: it may load, but the engine's own buffers or a longer context can push it over. "No" means it won't fit on the GPU alone. The 90% line is our own rule of thumb, not a vendor figure; for this model NVIDIA's recipe budgets (above) are the better guide. The "NVIDIA recipe" column adds the projector and drafter files. On a Mac, macOS also needs some of the unified memory, so treat Mac rows as optimistic.

Our recommendations

These follow from the file sizes, NVIDIA's recipe table and the 90% rule.^[2][4][3]

llama.cpp, Ollama and LM Studio support

• llama.cpp: NVIDIA's card names llama.cpp as the runtime and says to build the latest version from source.^[2] In llama.cpp's main branch (checked at commit bee23322, November 3, 2026), the GGUF's nemotron_h_moe architecture is supported and the draft-dflash speculative-decoding type used in NVIDIA's commands is present.^[16][17] Older builds may not load the model or the drafter.
• Ollama: the nemotron-3.5-bolt:74b-a9b tag (same as latest and 74b-a9b-iq2) is listed at 25 GB with a 256K context and text and image input.^[19]
• LM Studio: lmstudio-community publishes the same IQ2_XXS file (22.0 GB) and a 1.73 GB BF16 projector.^[20]

If you're short on memory, drop --mmproj (no image input) and the drafter (no speculative decoding) first: together they are 2.7 GB.^[4][14] With only 9B active parameters, llama.cpp can also keep some experts in system RAM; see our MoE expert offload guide. We have not measured its speed.

Cheapest GPUs to rent for Nemotron 3.5 Bolt 74B-A9B

On-demand prices from Runpod's pricing page for single GPUs that meet NVIDIA's 32 GB+ guidance (checked November 3, 2026; Community Cloud is cheaper but runs on shared hosts):^[13]