Best local AI models for the RTX 3090
24 GB of graphics memory and 936 GB/s of bandwidth, for about $1,250. The most capable model that fits with 32K context is Qwen3.8 27B (capability 152), at about 41 tokens a second. 7 of 14 models fit.
Every model on the RTX 3090
Usual quantisation
| Model | Capability | 8K | 32K | 128K | Speed at 32K |
|---|---|---|---|---|---|
| Qwen3.8 27BQ4_K_M, 20 GB at 32K | 152 | Fits | Fits | Partly, with system RAM | 41 tok/s |
| Gemma 4 31BQ4_K_M, 23 GB at 32K | 143 | Fits | Fits | Partly, with system RAM | 36 tok/s |
| Gemma 4 26B-A4BQ4_K_M, 18 GB at 32K | 142 | Fits | Fits | Fits | 138 tok/s |
| Qwen3.5 9BQ4_K_M, 7.6 GB at 32K | 139 | Fits | Fits | Fits | 107 tok/s |
| gpt-oss-20bMXFP4 (gpt-oss native), 13 GB at 32K | 138 | Fits | Fits | Fits | 167 tok/s |
| Nemotron 3.5 Lightning 30B-A3BQ4_K_M, 20 GB at 32K | — | Fits | Fits | Fits | 184 tok/s |
| Muse Glimmer 30BQ4_K_M, 23 GB at 32K | — | Fits | Fits | Partly, with system RAM | 36 tok/s |
| Qwen3.6 35B-A3BQ4_K_M, 23 GB at 32K | 144 | Fits | Partly, with system RAM | Partly, with system RAM | 105 tok/s |
| gpt-oss-120bMXFP4 (gpt-oss native), 66 GB at 32K | 140 | Partly, with system RAM | Partly, with system RAM | Partly, with system RAM | 22 tok/s |
| Llama 3.3 70BQ4_K_M, 55 GB at 32K | 127 | Partly, with system RAM | Partly, with system RAM | Won’t fit | 1.8 tok/s |
| Mistral Small 4 119B-A6BQ4_K_M, 77 GB at 32K | — | Partly, with system RAM | Partly, with system RAM | Partly, with system RAM | 19.1 tok/s |
| Qwen3.5 122B-A10BQ4_K_M, 77 GB at 32K | — | Partly, with system RAM | Partly, with system RAM | Partly, with system RAM | 12.0 tok/s |
| Nemotron 3 Super 120B-A12BQ4_K_M, 76 GB at 32K | — | Partly, with system RAM | Partly, with system RAM | Partly, with system RAM | 10.5 tok/s |
| Qwen3.5 397B-A17BQ4_K_M, 248 GB at 32K | 147 | Won’t fit | Won’t fit | Won’t fit | — |
Estimates from memory size and bandwidth, calibrated against published llama.cpp results. Capability: Epoch Capabilities Index by Epoch AI, CC BY 4.0, read 4 Nov 2026.
Similar-priced hardware
| Hardware | Memory | Bandwidth | Price (about) | Best model at 32K |
|---|---|---|---|---|
| RTX 3090 (used) | 24 GB | 936 GB/s | $1,250 | Qwen3.8 27B (152) |
| RTX 5080 | 16 GB | 960 GB/s | $1,600 | Qwen3.5 9B (139) |
| RTX 5070 Ti | 16 GB | 896 GB/s | $1,200 | Qwen3.5 9B (139) |
| RTX 5060 Ti 16 GB | 16 GB | 448 GB/s | $800 | Qwen3.5 9B (139) |
| Radeon RX 9070 XT | 16 GB | 640 GB/s | $800 | Qwen3.5 9B (139) |
Questions
What is the best local LLM for the RTX 3090?
The highest-scoring model that fits with 32K tokens of context is Qwen3.8 27B (capability 152), at an estimated 41 tokens a second. For speed, Nemotron 3.5 Lightning 30B-A3B runs at about 184 tokens a second.
How many models fit on the RTX 3090?
7 of the 14 models we track fit in its 24 GB with 32K context, and 8 with 8K. 4 fit with 128K tokens of context for long documents.
What does the RTX 3090 cost to run?
At about 310 W under load and $0.25 per kWh, about $0.08 an hour of generation. Against the API price of Qwen3.8 27B, at 4M input and 1M output tokens a day it would pay for itself in about 54 months.