Mixtral 8x7B VRAM requirements: can you run it locally?

Mixtral 8x7B Instruct v0.1 is Mistral AI's mixture-of-experts model (2 of 8 experts per token). Hugging Face lists 46,702,792,704 parameters, the repo was created on 2023-12-10, and the license is Apache 2.0.^[1] Its config.json describes the architecture as: sparse MoE: 32 layers, 8 experts with 2 routed per token, grouped-query attention (8 KV heads); the maximum context is 32,768 tokens.^[2] Hugging Face counts about 275,282 downloads in the last 30 days (checked October 31, 2026).^[1]

Memory needed at each quantization

Quantization stores each weight in fewer bits. Smaller files fit on smaller GPUs, at some cost in quality, and the cost grows below about 4 bits. Below is every GGUF build in the TheBloke/Mixtral-8x7B-Instruct-v0.1-GGUF repo, using the exact file sizes on Hugging Face (1 GB = 1,000,000,000 bytes; split files added together), largest first.^[3] There is no 16-bit GGUF in that repo, so the 16-bit row is the parameter count × 2 bytes.^[1]

Which quant should you pick? Q4_K_M is the usual balance of size and quality. Use Q5_K_M or Q6_K if you have the memory, and Q8_0 if you want results close to the original. Treat 2-bit and 3-bit builds as a last resort.

Context length adds memory (the KV cache)

On top of the weights, the engine keeps a key/value cache for every token. From config.json (32 layers, 8 KV heads, head size 128),^[2] an FP16 cache takes 2 × 32 × 8 × 128 × 2 bytes = 131,072 bytes per token.

Some engines can store the cache in 8-bit to roughly halve it.

Which GPUs and Macs it fits on

Each cell compares the weights plus an 8K-token KV cache with the device's memory.^{[4][6][7][8][9][10][11]}

Column headers show weights + 8,192-token KV cache.

How to read it: "Fits" means weights + 8K cache use under 90% of memory. "Tight" means 90–100%: it may load, but the engine's own working memory or a longer context can push it over. "No" means it won't fit on the GPU alone. The 90% line is our own rule of thumb, not a vendor figure. On a Mac, macOS also needs some of the unified memory, so treat Mac rows as optimistic.

Our recommendations

These follow from the file sizes and the 90% rule.^[3]

How to run it

With llama.cpp, LM Studio or Ollama, download a GGUF file from the quant list above (for example the Q4_K_M build) from the TheBloke/Mixtral-8x7B-Instruct-v0.1-GGUF repo.^[3] If the model is bigger than your VRAM, these tools can keep some layers in system RAM, but every token then waits on the slower memory, so expect much lower speed.

Cheapest GPU to rent for Mixtral 8x7B

On-demand prices listed on Runpod's pricing page for cards that hold Q4_K_M with an 8K context (checked October 31, 2026; Community Cloud is cheaper but runs on shared hosts):^[12]

Prices checked on 2026-10-31. They change often, so confirm on the provider's page. Each of these holds Q4_K_M plus an 8K context (27.5 GB) under our 90% rule. Rent on Runpod Referral link

See all GPU rental prices for Lambda, Vast.ai and multi-GPU setups, or compare it with other models in the VRAM table.^[13]

Frequently asked questions

How much VRAM does Mixtral 8x7B need at 4-bit?

About 27.5 GB for the Q4_K_M weights (26.4 GB) plus an 8K-token FP16 KV cache (1.07 GB), before engine overhead.^[3][2]

What is the smallest GPU that runs Mixtral 8x7B?

In our list, a 32 GB card (RTX 5090) holds Q4_K_M with an 8K context under our 90% rule. Smaller cards need a lower-bit quant from the table above.^[3]

How much does a 32K context add?

At FP16 the KV cache for 32,768 tokens is about 4.3 GB.^[2]

How we calculated

• Weights are the exact GGUF file sizes on Hugging Face, checked October 31, 2026.^[3]
• KV cache is calculated from config.json. We assume an FP16 cache.^[2]
• Device memory is from NVIDIA and Apple spec pages. We compare in decimal GB, so the results lean conservative.
• Rental prices are the listed on-demand Runpod rates, checked October 31, 2026.^[12]
• Not included: engine working buffers, the OS and display. These vary by engine and settings, so leave headroom.

Sources

All sources checked on 2026-10-31.

1. Hugging Face, mistralai/Mixtral-8x7B-Instruct-v0.1 (parameter count, license, repo date, 30-day downloads; checked October 31, 2026). https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1
2. Hugging Face, mistralai/Mixtral-8x7B-Instruct-v0.1 config.json (layers, KV heads, head size, context). https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1/blob/main/config.json
3. Hugging Face, TheBloke/Mixtral-8x7B-Instruct-v0.1-GGUF file list (exact GGUF sizes; checked October 31, 2026). https://huggingface.co/TheBloke/Mixtral-8x7B-Instruct-v0.1-GGUF/tree/main
4. NVIDIA, GeForce RTX 4060 family (8 GB / 16 GB). https://www.nvidia.com/en-us/geforce/graphics-cards/40-series/rtx-4060-4060ti/
5. NVIDIA, GeForce RTX 3060 family (12 GB). https://www.nvidia.com/en-us/geforce/graphics-cards/30-series/rtx-3060-3060ti/
6. NVIDIA, GeForce RTX 3090 specs (24 GB). https://www.nvidia.com/en-us/geforce/graphics-cards/30-series/rtx-3090-3090ti/
7. NVIDIA, GeForce RTX 4090 specs (24 GB). https://www.nvidia.com/en-us/geforce/graphics-cards/40-series/rtx-4090/
8. NVIDIA, GeForce RTX 5090 specs (32 GB). https://www.nvidia.com/en-us/geforce/graphics-cards/50-series/rtx-5090/
9. Apple, MacBook Pro / MacBook Air tech specs (unified memory options). https://www.apple.com/macbook-pro/specs/
10. NVIDIA, RTX 6000 Ada Generation (48 GB). https://www.nvidia.com/en-us/products/workstations/rtx-6000/
11. NVIDIA, A100 (80 GB). https://www.nvidia.com/en-us/data-center/a100/
12. Runpod, GPU pricing (checked October 31, 2026). https://www.runpod.io/pricing
13. VRAMNeeded comparison table (sources listed per model). https://vramneeded.com/compare/