Local AI: which models run on your hardware

14 open-weight models and 16 graphics cards and computers. Pick your hardware and what you’ll use it for to see which models fit in memory, how fast they’ll run and how capable they are. The most capable model here, Qwen3.8 27B, scores 152 on Epoch AI’s index.

Open-weight models on the chosen hardware: capability, memory needed, whether they fit and estimated speed
Model	Capability	Memory needed	Fits?	Speed (estimate)	Through an API, per 1M
Qwen3.8 27B27B · Q4_K_M	152	20 GB	Fits	44 tok/s	$0.039 / $1.25
Gemma 4 31B30.7B · Q4_K_M	143	23 GB	Fits	39 tok/s	$0.09 / $0.34
Gemma 4 26B-A4B25.2B, 3.8B active · Q4_K_M	142	18 GB	Fits	145 tok/s	$0.042 / $0.22
Qwen3.5 9B9B · Q4_K_M	139	7.6 GB	Fits	115 tok/s	$0.10 / $0.15
gpt-oss-20b21B, 3.6B active · MXFP4 (gpt-oss native)	138	13 GB	Fits	176 tok/s	$0.018 / $0.09
Nemotron 3.5 Lightning 30B-A3B30B, 3B active · Q4_K_M	—	20 GB	Fits	191 tok/s	$0.025 / $0.12
Muse Glimmer 30B30B · Q4_K_M	—	23 GB	Fits	39 tok/s	$0.30 / $1.20
Qwen3.6 35B-A3B35B, 3B active · Q4_K_M	144	23 GB	Partly, with system RAM	107 tok/s	$0.05 / $0.70
gpt-oss-120b117B, 5.1B active · MXFP4 (gpt-oss native)	140	66 GB	Partly, with system RAM	22 tok/s	$0.03 / $0.15
Llama 3.3 70B70.6B · Q4_K_M	127	55 GB	Partly, with system RAM	1.8 tok/s	$0.10 / $0.32
Mistral Small 4 119B-A6B119B, 6B active · Q4_K_M	—	77 GB	Partly, with system RAM	19.2 tok/s	$0.15 / $0.60
Qwen3.5 122B-A10B122B, 10B active · Q4_K_M	—	77 GB	Partly, with system RAM	12.1 tok/s	$0.26 / $2.08
Nemotron 3 Super 120B-A12B120B, 12B active · Q4_K_M	—	76 GB	Partly, with system RAM	10.5 tok/s	$0.085 / $0.40
Qwen3.5 397B-A17B397B, 17B active · Q4_K_M	147	248 GB	Won’t fit	—	$0.55 / $3.50

Showing RTX 4090 (used) with 32K tokens of context. Memory needed covers the model at its usual quantisation plus the context cache; speed is generation speed in tokens a second, estimated from memory bandwidth. Capability: Epoch Capabilities Index by Epoch AI, CC BY 4.0, read 4 Nov 2026. API prices are the cheapest host we track, per 1M input / output tokens.

The best model for each device

32K context

For each device, the highest-scoring model that fits in memory with 32K tokens of context
Hardware	Memory	Best model that fits	Speed	Price (about)
RTX 5090NVIDIA	32 GB	Qwen3.8 27Bcapability 152	75 tok/s	$5,500
RTX 5080NVIDIA	16 GB	Qwen3.5 9Bcapability 139	110 tok/s	$1,600
RTX 5070 TiNVIDIA	16 GB	Qwen3.5 9Bcapability 139	103 tok/s	$1,200
RTX 5060 Ti 16 GBNVIDIA	16 GB	Qwen3.5 9Bcapability 139	55 tok/s	$800
RTX 4090 (used)NVIDIA	24 GB	Qwen3.8 27Bcapability 152	44 tok/s	$2,250
RTX 3090 (used)NVIDIA	24 GB	Qwen3.8 27Bcapability 152	41 tok/s	$1,250
RTX 3060 12 GBNVIDIA	12 GB	Qwen3.5 9Bcapability 139	45 tok/s	$300
RTX PRO 6000 BlackwellNVIDIA	96 GB	Qwen3.8 27Bcapability 152	75 tok/s	$9,000
Radeon RX 9070 XTAMD	16 GB	Qwen3.5 9Bcapability 139	76 tok/s	$800
Radeon RX 7900 XTXAMD	24 GB	Qwen3.8 27Bcapability 152	42 tok/s	$1,130
Radeon AI PRO R9700AMD	32 GB	Qwen3.8 27Bcapability 152	30 tok/s	$1,650
Intel Arc Pro B70Intel	32 GB	Qwen3.8 27Bcapability 152	21 tok/s	$949
Mac Studio M5 Max 128 GBApple	128 GB	Qwen3.8 27Bcapability 152	27 tok/s	$4,500
Mac Studio M5 Ultra 512 GBApple	512 GB	Qwen3.8 27Bcapability 152	34 tok/s	$10,500
Ryzen AI Max+ 395 128 GBAMD	128 GB	Qwen3.8 27Bcapability 152	10.9 tok/s	$3,300
NVIDIA DGX SparkNVIDIA	128 GB	Qwen3.8 27Bcapability 152	11.6 tok/s	$4,699

Hardware prices are typical street prices we use in the calculator, not quotes. Used cards are marked.

The models

Local or API?

Memory decides what runs. A model has to fit in your graphics card’s memory (or a Mac’s unified memory) with room for its context. If it doesn’t, part of it moves to system RAM and speed drops sharply.
Bandwidth decides speed. Generating each token reads the model’s active weights, so memory bandwidth sets the pace. Mixture-of-experts models such as gpt-oss-120b read only a fraction of their weights per token, so they run faster than their size suggests.
APIs are cheap for open models. The same models cost cents per million tokens through hosts, so a graphics card pays for itself only with heavy daily use. Work out your own break-even in the local LLM calculator.
Privacy and control. Nothing leaves your machine, it works offline, and the cost doesn’t grow with use.

Questions

What is the best local LLM for a 24GB graphics card?

On an RTX 4090 or 3090 (24GB), the highest-scoring model that fits with 32K tokens of context is Qwen3.8 27B (capability 152), at an estimated 44 tokens a second on the 4090.

What is the best local LLM for 16GB of VRAM?

Qwen3.5 9B (capability 139) fits on a 16GB card such as the RTX 5060 Ti with 32K tokens of context, at about 55 tokens a second.

Can I run a local LLM on 12GB of VRAM?

Yes, for shorter conversations: Qwen3.5 9B fits on an RTX 3060 12GB with 8K tokens of context, at about 50 tokens a second. Larger models spill into system memory and slow down sharply.

What can a Mac Studio run?

A Mac Studio M5 Max with 128GB fits 13 of the 14 models here with 32K tokens of context. The highest-scoring one that fits is Qwen3.8 27B, at about 27 tokens a second.

How good are local models compared with ChatGPT or Claude?

The best-scoring open model here, Qwen3.8 27B, scores 152 on Epoch AI’s Capabilities Index, close to GLM-5.3 Flash (153). The top API models score up to 169, so the gap is largest on the hardest tasks.

Are these speeds measured?

They’re estimates from each device’s memory bandwidth, calibrated against published llama.cpp benchmarks, the same method as our local LLM calculator. Real speeds vary with software, drivers and settings, typically within about a quarter either way.

Is running AI locally cheaper than an API?

Only with heavy, steady use. Open models are cheap through hosted APIs, often cents per million tokens, so a graphics card can take years to pay for itself. Privacy, offline use and fixed costs are the stronger reasons to run locally.

Models for your hardware

The best model for each device

The models

Local or API?

Questions