Measured tokens per second by GPU

VRAM tells you whether a model fits. This page tells you how fast it runs once it does. Every number below was measured and published by someone else, and each row links to where it came from. We don't estimate speeds or scale them up from other cards. If no one has published a result, the card isn't listed.

How to read it: Generation is how fast the answer appears, in tokens per second (about 0.75 words per token). Anything over about 10 tok/s feels like reading speed. Prompt is how fast the model reads what you paste in, which matters for long documents.

Sources

1. llama.cpp GitHub: Performance of llama.cpp on Nvidia CUDA (#15013), scoreboard, posted 2025-08-01. Test: llama-bench pp512 / tg128, no flash attention, batch default.
2. llama.cpp GitHub: Performance of llama.cpp on AMD ROCm (HIP) (#15021), scoreboard, posted 2025-08-01. Test: llama-bench pp512 / tg128.
3. llama.cpp GitHub: Performance of llama.cpp with Vulkan (#10879), scoreboard, posted 2024-12-18. Test: llama-bench pp512 / tg128.
4. llama.cpp GitHub: Performance of llama.cpp on Apple Silicon M-series (#4167), summary table, posted 2023-11-22. Test: llama-bench pp512 / tg128.
5. XiongjieDai/GPU-Benchmarks-on-LLM-Inference (GitHub README), Llama 3 tables, posted 2024-05-13. Test: llama.cpp main, average over generating 1024 tokens; prompt eval averaged over 1024 tokens.

Why results differ

• Test length: the llama.cpp scoreboards use a short 512-token prompt and 128 generated tokens. Long chats and big context windows are slower.
• Model and quant: most published scoreboards use Llama 2 7B at Q4_0 so cards can be compared fairly. Bigger models run slower, roughly in line with their file size.
• Engine and version: llama.cpp, Ollama (which runs llama.cpp), vLLM and MLX all perform differently, and new releases often get faster.

Got a result we're missing?

Every number above comes from a public, linked benchmark. We don't take reader submissions on this page, and we don't fill gaps with estimates. If you've posted a llama-bench run publicly, it may be added once we can check it against its source.

Want to check a card's VRAM first? Use the VRAM calculator or the VRAM table.