VRAMNeeded Calculator

Run a model bigger than your VRAM with MoE expert offload

Short answer: if the model is a Mixture of Experts (MoE), you don't need the whole thing in VRAM. Each token only uses a few experts, so you can keep the attention layers and KV cache on the GPU and store the expert weights in system RAM. llama.cpp does this with --cpu-moe or --n-cpu-moe N. The model then needs enough VRAM for the non-expert layers and the context, plus enough RAM for the experts.

When it works and when it doesn't

  • Works: MoE models such as Qwen3.8-35B-A3B, Qwen3.8-Flash-Next, Flint and DeepSeek-V4.1-Flash, where only a small share of the weights is active per token.
  • Doesn't help: dense models (every weight is used on every token). For those, plain layer offload with a lower -ngl is the option, and it's slower. See fix 8 in How to fix "CUDA out of memory".
  • Speed depends on system RAM, not just the GPU. Experts read from RAM are slower than from VRAM, so expect lower speeds than a full-VRAM load.
  • You still need the RAM. Weights that don't fit in VRAM or RAM will not load.

Reported setups

These are community reports from single machines, not our measurements. Your speed will differ.

Model GPU VRAM + system RAM Reported result Source
Qwen3.8-35B-A3B, Q4_K_M (21.7 GB of weights) 12 GB (RTX 3060) + 16 GB ~50 tok/s decode, --n-cpu-moe 26 (self-reported) @DogukanUrker
Flint 40B-A2.5B 12 GB (RTX 3060) + 16 GB Runs on one card (self-reported) @Oluwaphilemon1
DeepSeek-V4.1-Flash, 2-bit 16 GB (RTX 2000 Ada) + 64 GB 7.0 tok/s warm, short answers (self-reported) @TheTokenVoyager
Qwen3.8-Flash-Next Coder, 3.5 BPW 8 GB + 24 GB ~8 tok/s generation, selective expert offload (self-reported) @CNopenmodels

Check the exact file size on the model page before you download. Our VRAM calculator shows the total size at each quantization.

The settings

llama.cpp (llama-server, llama-cli)

Flag What it does
-ngl, --n-gpu-layers N Max layers stored in VRAM (auto, all or a number).
-cmoe, --cpu-moe Keep all MoE expert weights on the CPU.
-ncmoe, --n-cpu-moe N Keep the MoE weights of the first N layers on the CPU.
-ot, --override-tensor <pattern>=<buffer type> Choose where matching tensors go, for finer control than --n-cpu-moe.
-ctk / -ctv q8_0 Quantize the K and V cache (default is f16).
-fa, --flash-attn on Turn Flash Attention on (default auto).
-c, --ctx-size N Context length. Smaller means less KV cache.

A starting point:

llama-server -m model.gguf -ngl all --n-cpu-moe 99 -c 16384 -fa on -ctk q8_0 -ctv q8_0

LM Studio

In the model's advanced load settings, turn on the option to force MoE expert weights onto the CPU (added in LM Studio 0.3.23). LM Studio describes it as a setting to "try this for low VRAM machines". It places all experts on the CPU; there is no per-layer count.

Ollama

Ollama splits a model between GPU and CPU automatically when it doesn't fit, but its documentation has no MoE-expert-specific setting. You can still shrink the KV cache with OLLAMA_KV_CACHE_TYPE=q8_0 and OLLAMA_FLASH_ATTENTION=1. For expert offload, use llama.cpp or LM Studio.

Tune it step by step

  1. Start with all experts on the CPU (--n-cpu-moe set to the model's layer count, or --cpu-moe) and -ngl all.
  2. Watch nvidia-smi while it runs. If there's VRAM to spare, lower N by a few layers so more experts sit on the GPU.
  3. Stop lowering N when you get an out-of-memory error, then go back up one step.
  4. If you need longer context, trade some of that VRAM back with a smaller -c or a q8_0 KV cache.

Check it isn't spilling

  • Ollama: ollama ps shows the CPU/GPU split for each loaded model.
  • llama.cpp: nvidia-smi shows how much VRAM the process holds.

By card size

Still too slow?

We may earn from links marked Affiliate partner (disclosure).

FAQ

Does expert offload lower quality? No. The weights are the same; only where they're stored changes. Speed is what drops.

Does it work for dense models? No. Use a smaller quantization or -ngl layer offload instead.

--cpu-moe or --n-cpu-moe? --cpu-moe puts all experts on the CPU. --n-cpu-moe N does it only for the first N layers, so you can fill spare VRAM.

Sources

  • llama.cpp: server README (-ngl, --cpu-moe, --n-cpu-moe, -ot, -ctk/-ctv, -fa, -c)
  • LM Studio: 0.3.23 release notes
  • Ollama: FAQ
  • Community reports linked in the table above (self-reported, X)

Last checked: