Best local AI models for the RTX 3060 12 GB
12 GB of graphics memory and 360 GB/s of bandwidth, for about $300. The most capable model that fits with 32K context is Qwen3.5 9B (capability 139), at about 45 tokens a second. 1 of 14 models fit.
Every model on the RTX 3060 12 GB
Usual quantisation
| Model | Capability | 8K | 32K | 128K | Speed at 32K |
|---|---|---|---|---|---|
| Qwen3.5 9BQ4_K_M, 7.6 GB at 32K | 139 | Fits | Fits | Fits | 45 tok/s |
| Qwen3.8 27BQ4_K_M, 20 GB at 32K | 152 | Partly, with system RAM | Partly, with system RAM | Partly, with system RAM | 5.5 tok/s |
| Qwen3.6 35B-A3BQ4_K_M, 23 GB at 32K | 144 | Partly, with system RAM | Partly, with system RAM | Partly, with system RAM | 35 tok/s |
| Gemma 4 31BQ4_K_M, 23 GB at 32K | 143 | Partly, with system RAM | Partly, with system RAM | Partly, with system RAM | 4.4 tok/s |
| Gemma 4 26B-A4BQ4_K_M, 18 GB at 32K | 142 | Partly, with system RAM | Partly, with system RAM | Partly, with system RAM | 31 tok/s |
| gpt-oss-120bMXFP4 (gpt-oss native), 66 GB at 32K | 140 | Partly, with system RAM | Partly, with system RAM | Partly, with system RAM | 16.9 tok/s |
| gpt-oss-20bMXFP4 (gpt-oss native), 13 GB at 32K | 138 | Partly, with system RAM | Partly, with system RAM | Partly, with system RAM | 49 tok/s |
| Llama 3.3 70BQ4_K_M, 55 GB at 32K | 127 | Partly, with system RAM | Partly, with system RAM | Won’t fit | 1.3 tok/s |
| Nemotron 3.5 Lightning 30B-A3BQ4_K_M, 20 GB at 32K | — | Partly, with system RAM | Partly, with system RAM | Partly, with system RAM | 44 tok/s |
| Muse Glimmer 30BQ4_K_M, 23 GB at 32K | — | Partly, with system RAM | Partly, with system RAM | Partly, with system RAM | 4.4 tok/s |
| Qwen3.5 397B-A17BQ4_K_M, 248 GB at 32K | 147 | Won’t fit | Won’t fit | Won’t fit | — |
| Mistral Small 4 119B-A6BQ4_K_M, 77 GB at 32K | — | Won’t fit | Won’t fit | Won’t fit | — |
| Nemotron 3 Super 120B-A12BQ4_K_M, 76 GB at 32K | — | Won’t fit | Won’t fit | Won’t fit | — |
| Qwen3.5 122B-A10BQ4_K_M, 77 GB at 32K | — | Won’t fit | Won’t fit | Won’t fit | — |
Estimates from memory size and bandwidth, calibrated against published llama.cpp results. Capability: Epoch Capabilities Index by Epoch AI, CC BY 4.0, read 4 Nov 2026.
Questions
What is the best local LLM for the RTX 3060 12 GB?
The highest-scoring model that fits with 32K tokens of context is Qwen3.5 9B (capability 139), at an estimated 45 tokens a second.
How many models fit on the RTX 3060 12 GB?
1 of the 14 models we track fit in its 12 GB with 32K context, and 1 with 8K. 1 fit with 128K tokens of context for long documents.
What does the RTX 3060 12 GB cost to run?
At about 190 W under load and $0.25 per kWh, about $0.05 an hour of generation. Against the API price of Qwen3.5 9B, at 4M input and 1M output tokens a day it would pay for itself in about 57 months.