Can I run this LLM locally?
Pick a model and your GPU or Mac. See how much memory it needs at your quantization and context length — and what to change if it doesn’t fit.
Result
Fits fully
Qwen3 8B · Q4_K_M · 8K → NVIDIA GeForce RTX 3060 12GB. 4.52 GB of headroom.
| Component | Memory |
|---|---|
| Weightsactual GGUF file: Q4_K_M | 4.68 GB |
| KV cache8K tokens × 1 seq, F16 | 1.13 GB |
| Runtime overhead10% of weights | 0.47 GB |
| Total needed | 6.28 GB |
| Usable on your hardware90% of 12.0 GB | 10.8 GB |
Run it
Starting point, not a guarantee. Downloads the GGUF from Hugging Face.
llama-server -hf unsloth/Qwen3-8B-GGUF:Q4_K_M -c 8192 -ngl 999OLLAMA_CONTEXT_LENGTH=8192 ollama serve
ollama run hf.co/unsloth/Qwen3-8B-GGUF:Q4_K_MModel details
- Parameters
- 8.2BDerived from the BF16 GGUF size (16388044384 bytes / 2 bytes per weight), text model only
- Layers
- 36
- Attention
- Full attention (GQA)36× full (8 KV × 128)
- Max context
- 40K
- License
- apache-2.0
- Config source
- Qwen/Qwen3-8Bsynced 2026-10-18
How the estimate works
- Weights: the real GGUF file size from Hugging Face when available, otherwise parameters × bits per weight ÷ 8.
- KV cache: computed from the model’s real architecture — KV heads (GQA), sliding-window layers, MLA and hybrid linear-attention layers.
- Overhead and usable memory: 10% of weights and ~90% of VRAM (~72% on Apple Silicon) by default. Every assumption is adjustable.
Frequently asked questions
How much VRAM do I need to run an LLM locally?
Roughly the model file size plus the KV cache plus about 10% runtime overhead. At the popular Q4_K_M quantization a model needs about 0.6 GB per billion parameters for weights (an 8B model ≈ 5 GB, a 70B model ≈ 43 GB), and the KV cache adds more as the context grows. Use the calculator for your exact model and context.
Does context length use VRAM?
Yes. Every token in the context stores keys and values for each attention layer. For Llama 3.1 8B at FP16 that is 1 GiB per 8,192 tokens; at 128K tokens it is 16 GiB — more than the weights. Models with sliding-window, MLA or hybrid linear attention need much less. Quantizing the KV cache to Q8_0 roughly halves it.
What happens if a model does not fit in my GPU?
llama.cpp, Ollama and LM Studio can keep some layers on the GPU and the rest in system RAM (partial offload). It works, but generation is usually much slower because the CPU part becomes the bottleneck. The calculator estimates how many layers fit (the -ngl value).
Can I combine two GPUs?
Yes. llama.cpp splits layers across GPUs, so their memory adds up, minus some unused space on each card. Add a second GPU in the calculator; the split loss is adjustable under Assumptions.
Why can my Mac not use all of its memory for the model?
Apple Silicon shares memory between the CPU, GPU and macOS, and by default only part of it can be wired for the GPU. A practical rule is about 70–75% of the total for the model, which is the calculator’s default for Macs.