VRAMNeeded Calculator

Fix llama.cpp "failed to allocate CUDA0 buffer"

These are the lines llama.cpp prints when the GPU runs out of memory while loading a model. This one is quoted from llama.cpp issue #19437 (GTX 1080 Ti, 11 GB):

ggml_backend_cuda_buffer_type_alloc_buffer: allocating 9814.99 MiB on device 0: cudaMalloc failed: out of memory
alloc_tensor_range: failed to allocate CUDA0 buffer of size 10291759104
llama_model_load: error loading model: unable to allocate CUDA0 buffer
llama_model_load_from_file_impl: failed to load model

The sizes differ from case to case. On AMD (ROCm) the device shows as ROCm0 instead of CUDA0.

What it means

llama.cpp tried to reserve one block of VRAM and the driver said no. The block is one of three things:

  • Model weights for the layers you put on the GPU (-ngl).
  • KV cache for the context size (-c). At a model's full native context this can be many gigabytes.
  • Compute buffers. These grow with the batch size, and without flash attention they can be very large at long context.

Fixes, in order

1. Don't force all layers onto the GPU

Newer llama.cpp builds can fit settings to your free VRAM automatically, but only for parameters you didn't set yourself. In issue #19437 the log shows failed to fit params to free device memory: n_gpu_layers already set by user to 99, abort, and a maintainer replied that because the user told it to use up to 99 layers, the autotuner won't override that. Remove -ngl 99, or lower it until the model loads. The layers that don't fit run from system RAM, which is slower but works.

2. Set a smaller context

-c 8192 (--ctx-size) instead of the model's maximum. The KV cache scales with context, so this is often the biggest saving. In the same issue, llama.cpp's own fitter cut the context from 262144 to 30000 to save 6257 MiB.

3. Turn on flash attention and lower the batch size

In issue #10052 a 1B model failed with allocating 8849.72 MiB on device 0: cudaMalloc failed: out of memory. A maintainer explained that without flash attention the KQ matrix has to sit in VRAM and can be very large at long context, and suggested enabling flash attention (-fa), lowering -c, or lowering the micro-batch size (-ub).

4. Quantize the KV cache

--cache-type-k q8_0 --cache-type-v q8_0 roughly halves KV cache memory compared with 16-bit.

5. Use a smaller quant

A Q4_K_M GGUF is about half the size of Q8_0. Check how much each quant needs in the VRAM calculator.

6. For MoE models, offload only the experts

Keep attention on the GPU and park expert weights in RAM. See MoE expert offload.

Check first, or rent more VRAM

What can I run? lists which models fit your card. If it still won't fit, our GPU prices page compares hourly rental prices for 24, 48 and 80 GB cards (we may earn from some links, disclosure). For PyTorch errors, see Fix CUDA out of memory.

Sources

Last checked: