VRAMNeeded Calculator

How to fix "CUDA out of memory"

If you're reading this, you probably just saw something like this:

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB.
GPU 0 has a total capacity of 23.65 GiB of which 1.12 GiB is free. Of the allocated memory 20.84 GiB is allocated by PyTorch, and 1.03 GiB
is reserved by PyTorch but unallocated. If reserved but unallocated memory
is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
to avoid fragmentation.

Older PyTorch versions word it a little differently, and the numbers will differ, but the meaning is the same: something asked the GPU for more memory than it had free. Below are the fixes, starting with the quickest.

Why it happens

GPU memory (VRAM) has to hold several things at once:

  • The model weights. About 2 bytes per parameter at 16-bit, 1 byte at 8-bit and roughly 0.5 bytes at 4-bit. A 7B model needs about 14 GB at 16-bit before anything else.
  • The KV cache when generating text. It grows with context length and with the number of conversations running at once.
  • Activations during training, which grow with batch size and sequence length.
  • Gradients and optimizer state when training. Adam keeps two extra values per parameter, so full fine-tuning needs several times the model's size.
  • Overhead: the CUDA context, the PyTorch caching allocator, and anything else running on the GPU, including your desktop and other processes.

The "Tried to allocate" number is just the one request that failed. The real problem is everything already in memory before it.

Fixes, fastest first

1. Free up memory held by other processes

Run nvidia-smi and look at the process list at the bottom. A crashed notebook, an old Python session or a forgotten Ollama server can hold gigabytes.

nvidia-smi
kill <PID>        # end the process that's holding memory

In Jupyter, restarting the kernel frees everything. torch.cuda.empty_cache() returns cached blocks that no tensor is using any more, but it can't free memory your code still references. Delete the tensors (del x) first.

2. Lower the batch size

For training and batch inference, halve the batch size until it fits. To keep the same effective batch size, use gradient accumulation: run several small batches and step the optimizer once (in Hugging Face Trainer, set gradient_accumulation_steps).

3. Shorten the context or sequence length

For LLMs, the KV cache can be bigger than you expect at long context. Set the context to what you actually need:

  • vLLM: --max-model-len 8192
  • llama.cpp: -c 8192 (--ctx-size)
  • Ollama: OLLAMA_CONTEXT_LENGTH=8192, or num_ctx in a Modelfile or API request

For training, truncate or bucket long sequences.

4. Run fewer requests in parallel

Each parallel conversation needs its own share of KV cache.

  • vLLM: lower --max-num-seqs. vLLM also reserves a fixed share of the GPU up front, set with --gpu-memory-utilization (a fraction from 0 to 1; most of the GPU by default). Lower it if something else shares the GPU, or raise it slightly if vLLM itself fails to fit its KV cache.
  • llama.cpp server: lower --parallel (-np). The total context is split between the slots.
  • Ollama: set OLLAMA_NUM_PARALLEL=1, and OLLAMA_MAX_LOADED_MODELS=1 so only one model stays loaded.

5. Use a smaller quantization

Going from 16-bit to 8-bit roughly halves the size of the weights, and 4-bit roughly quarters it, usually with a small loss in quality.

  • GGUF (llama.cpp, Ollama): pick a Q4_K_M or Q5_K_M file instead of Q8_0 or F16.
  • Transformers: load with bitsandbytes load_in_4bit=True or load_in_8bit=True.
  • vLLM: use an AWQ, GPTQ or FP8 checkpoint.

You can also quantize the KV cache: --cache-type-k q8_0 --cache-type-v q8_0 in llama.cpp, OLLAMA_KV_CACHE_TYPE=q8_0 (with OLLAMA_FLASH_ATTENTION=1) in Ollama, or --kv-cache-dtype fp8 in vLLM.

6. Turn on gradient checkpointing (training)

Gradient checkpointing saves only some activations and recomputes the rest during the backward pass. It cuts activation memory a lot, but training runs slower because the backward pass recomputes what was discarded. In Transformers, call model.gradient_checkpointing_enable() or pass gradient_checkpointing=True to TrainingArguments. For fine-tuning on one card, LoRA or QLoRA instead of full fine-tuning avoids storing optimizer state for every weight.

7. Reduce fragmentation with expandable_segments

If the error says a lot of memory is "reserved by PyTorch but unallocated", the memory is there but split into pieces that are too small. Set this before starting Python:

export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

This lets the PyTorch allocator grow memory segments instead of making new fixed-size ones, which helps when tensor sizes keep changing.

8. Offload part of the model to the CPU

Offloading runs when the model doesn't quite fit, but it's much slower because data has to cross the PCIe bus.

  • llama.cpp: lower -ngl (--n-gpu-layers) so some layers stay in system RAM.
  • Ollama: does this automatically when the model doesn't fit. ollama ps shows the CPU/GPU split.
  • Transformers / Accelerate: device_map="auto" with a max_memory limit.
  • vLLM: --cpu-offload-gb.

If the model is a Mixture of Experts (MoE), you can offload only the expert weights and keep the rest on the GPU. See MoE expert offload.

Size it before you run it

Most of these errors can be avoided by checking first. Our VRAM calculator shows how much memory a model needs at each quantization and context length, and What can I run? lists what fits on your card.

Or rent a bigger GPU

If the model won't fit even after quantizing, or offloading is too slow, renting a 24 GB, 48 GB or 80 GB card by the hour is often cheaper than buying one. Our GPU prices page compares current hourly rates across providers. We may earn from some of those links (disclosure).

Tool-specific errors

Sources

Last checked: