Best local AI models for the RTX 5090
32 GB of graphics memory and 1792 GB/s of bandwidth, for about $5,500. The most capable model that fits with 32K context is Qwen3.8 27B (capability 152), at about 75 tokens a second. 8 of 14 models fit.
Every model on the RTX 5090
Usual quantisation
| Model | Capability | 8K | 32K | 128K | Speed at 32K |
|---|---|---|---|---|---|
| Qwen3.8 27BQ4_K_M, 20 GB at 32K | 152 | Fits | Fits | Fits | 75 tok/s |
| Qwen3.6 35B-A3BQ4_K_M, 23 GB at 32K | 144 | Fits | Fits | Fits | 177 tok/s |
| Gemma 4 31BQ4_K_M, 23 GB at 32K | 143 | Fits | Fits | Fits | 68 tok/s |
| Gemma 4 26B-A4BQ4_K_M, 18 GB at 32K | 142 | Fits | Fits | Fits | 204 tok/s |
| Qwen3.5 9BQ4_K_M, 7.6 GB at 32K | 139 | Fits | Fits | Fits | 186 tok/s |
| gpt-oss-20bMXFP4 (gpt-oss native), 13 GB at 32K | 138 | Fits | Fits | Fits | 254 tok/s |
| Nemotron 3.5 Lightning 30B-A3BQ4_K_M, 20 GB at 32K | — | Fits | Fits | Fits | 240 tok/s |
| Muse Glimmer 30BQ4_K_M, 23 GB at 32K | — | Fits | Fits | Partly, with system RAM | 67 tok/s |
| gpt-oss-120bMXFP4 (gpt-oss native), 66 GB at 32K | 140 | Partly, with system RAM | Partly, with system RAM | Partly, with system RAM | 27 tok/s |
| Llama 3.3 70BQ4_K_M, 55 GB at 32K | 127 | Partly, with system RAM | Partly, with system RAM | Partly, with system RAM | 2.3 tok/s |
| Mistral Small 4 119B-A6BQ4_K_M, 77 GB at 32K | — | Partly, with system RAM | Partly, with system RAM | Partly, with system RAM | 23 tok/s |
| Qwen3.5 122B-A10BQ4_K_M, 77 GB at 32K | — | Partly, with system RAM | Partly, with system RAM | Partly, with system RAM | 14.0 tok/s |
| Nemotron 3 Super 120B-A12BQ4_K_M, 76 GB at 32K | — | Partly, with system RAM | Partly, with system RAM | Partly, with system RAM | 12.4 tok/s |
| Qwen3.5 397B-A17BQ4_K_M, 248 GB at 32K | 147 | Won’t fit | Won’t fit | Won’t fit | — |
Estimates from memory size and bandwidth, calibrated against published llama.cpp results. Capability: Epoch Capabilities Index by Epoch AI, CC BY 4.0, read 4 Nov 2026.
Similar-priced hardware
| Hardware | Memory | Bandwidth | Price (about) | Best model at 32K |
|---|---|---|---|---|
| RTX 5090 | 32 GB | 1792 GB/s | $5,500 | Qwen3.8 27B (152) |
| RTX 4090 (used) | 24 GB | 1008 GB/s | $2,250 | Qwen3.8 27B (152) |
| Mac Studio M5 Max 128 GB | 128 GB | 614 GB/s | $4,500 | Qwen3.8 27B (152) |
| Ryzen AI Max+ 395 128 GB | 128 GB | 256 GB/s | $3,300 | Qwen3.8 27B (152) |
| NVIDIA DGX Spark | 128 GB | 273 GB/s | $4,699 | Qwen3.8 27B (152) |
Questions
What is the best local LLM for the RTX 5090?
The highest-scoring model that fits with 32K tokens of context is Qwen3.8 27B (capability 152), at an estimated 75 tokens a second. For speed, gpt-oss-20b runs at about 254 tokens a second.
How many models fit on the RTX 5090?
8 of the 14 models we track fit in its 32 GB with 32K context, and 8 with 8K. 7 fit with 128K tokens of context for long documents.
What does the RTX 5090 cost to run?
At about 455 W under load and $0.25 per kWh, about $0.11 an hour of generation. Against the API price of Qwen3.8 27B, at 4M input and 1M output tokens a day it would pay for itself in about 197 months.