Run Mistral Small 4 119B-A6B locally: memory, speed and cost
Mistral Small 4 119B-A6B has 119 billion parameters, 6 billion active per token (mixture of experts). At Q4_K_M it needs about 77 GB with 32K tokens of context, so the cheapest hardware here that runs it is the Ryzen AI Max+ 395 128 GB (about 33 tokens a second). Through an API it costs $0.15 / $0.60 per 1M tokens.
Memory needed
Weights plus context cache
| Quantisation | 8K context | 32K context | 128K context |
|---|---|---|---|
| Q4_K_Musual choice | 75 GB | 77 GB | 83 GB |
| Q5_K_M | 88 GB | 89 GB | 96 GB |
| Q8_0 | 131 GB | 132 GB | 139 GB |
| BF16 | 244 GB | 246 GB | 252 GB |
Estimates: weights at the stated bits per weight, plus the context (KV) cache, plus about 1 GB of overhead. Lower-bit quantisations save memory at some cost in quality.
Mistral Small 4 119B-A6B on each device
Q4_K_M, 32K context
| Hardware | Memory | Fits? | Speed (estimate) |
|---|---|---|---|
| RTX 5090 | 32 GB | Partly, with system RAM | 23 tok/s |
| RTX 5080 | 16 GB | Partly, with system RAM | 17.1 tok/s |
| RTX 5070 Ti | 16 GB | Partly, with system RAM | 17.0 tok/s |
| RTX 5060 Ti 16 GB | 16 GB | Partly, with system RAM | 16.0 tok/s |
| RTX 4090 (used) | 24 GB | Partly, with system RAM | 19.2 tok/s |
| RTX 3090 (used) | 24 GB | Partly, with system RAM | 19.1 tok/s |
| RTX 3060 12 GB | 12 GB | Won’t fit | — |
| RTX PRO 6000 Blackwell | 96 GB | Fits | 164 tok/s |
| Radeon RX 9070 XT | 16 GB | Partly, with system RAM | 16.6 tok/s |
| Radeon RX 7900 XTX | 24 GB | Partly, with system RAM | 19.1 tok/s |
| Radeon AI PRO R9700 | 32 GB | Partly, with system RAM | 21 tok/s |
| Intel Arc Pro B70 | 32 GB | Partly, with system RAM | 19.3 tok/s |
| Mac Studio M5 Max 128 GB | 128 GB | Fits | 74 tok/s |
| Mac Studio M5 Ultra 512 GB | 512 GB | Fits | 89 tok/s |
| Ryzen AI Max+ 395 128 GB | 128 GB | Fits | 33 tok/s |
| NVIDIA DGX Spark | 128 GB | Fits | 34 tok/s |
Generation speed estimated from memory bandwidth, calibrated against published llama.cpp results. Fastest here: RTX PRO 6000 Blackwell, about 164 tokens a second. Try other settings in the calculator.
How capable is it?
Epoch AI hasn’t scored Mistral Small 4 119B-A6B yet, so we don’t show a capability figure.
Or use it through an API
$0.15 / $0.60 per 1M input / output tokens (Mistral Small 4 hosted prices). A customer chat assistant of 100M input and 20M output tokens a month costs about $27.00, so local hardware pays for itself only with heavy use.
Questions
How much VRAM does Mistral Small 4 119B-A6B need?
About 75 GB at Q4_K_M with 8K tokens of context, 77 GB with 32K and 83 GB with 128K. At 8-bit (Q8_0) it needs about 131 GB with 8K context.
Can Mistral Small 4 119B-A6B run on a Mac?
Yes. A Mac Studio M5 Max with 128GB runs it with 32K context at an estimated 74 tokens a second.
How fast is Mistral Small 4 119B-A6B on an RTX 4090?
It doesn’t fit in the 4090’s 24GB with 32K context, so part of it runs from system RAM at about 19.2 tokens a second.
What does Mistral Small 4 119B-A6B cost through an API?
$0.15 / $0.60 per 1M input / output tokens. A customer chat assistant of 100M input and 20M output tokens a month would cost about $27.00.