Graphics cards
Which AI models run on the RTX 5090 (32 GB)?
The RTX 5090 (32 GB) has 32 GB of memory, about 31 GB of it usable for a model, and 1792 GB/s of memory bandwidth. 86 of the 110 models in our catalog run on it at 4-bit or better.
Data as of 2 November 2026 · Vendor specs
32 GBmemory
~31 GB usable for a model
1792 GB/sbandwidth, sets the speed
86models run at 4-bit or better
Models that run on it
| Model | Best build | Memory | Speed |
|---|---|---|---|
| Qwen3.6-35B-A3B35B MoE | Q4_K_M | 24 GB | ~346 tok/s ModelFit est. |
| Gemma 4 31B30.7B | Q4_K_M | 21 GB | ~73 tok/s ModelFit est. |
| Qwen3-Coder-30B-A3B30.5B MoE | Q4_K_M | 20 GB | ~287 tok/s ModelFit est. |
| Nemotron 3 Nano 30B-A3B30B MoE | Q6_K | 26 GB | ~398 tok/s our est. |
| Qwen3.8-27B27.3B | Q8_0 | 29 GB | ~36 tok/s our est. |
| Gemma 4 26B-A4B25.2B MoE | Q8_0 | 30 GB | ~195 tok/s ModelFit est. |
| gpt-oss-20b21B MoE | MXFP4 | 15.1 GB | ~261 tok/s ModelFit est. |
| Nemotron Nano 9B v29B | FP16 | 19.4 GB | ~55 tok/s our est. |
| Qwen3.5 9B Instruct9B | Q8_0 | 12.0 GB | ~153 tok/s ModelFit est. |
| Qwen2.5 7B Instruct7B | Q4_K_M | 6.6 GB | ~240 tok/s ModelFit est. |
| Gemma 3 4B Instruct4B | Q8_0 | 5.0 GB | ~249 tok/s ModelFit est. |
| Flint 40B-A2.5B39.8B MoE | NVFP4 | 24 GB | ~363 tok/s ModelFit est. |
| Sage-A1.535.95B | Q4_K_M | 23 GB | ~346 tok/s ModelFit est. |
| Qwen3.5 35B-A3B Instruct35B MoE | Q4_K_M | 21 GB | ~350 tok/s ModelFit est. |
| Ornith 1.0 35B35B | Q4_K_M | 23 GB | ~350 tok/s ModelFit est. |
| Nanbeige4.3 33B33.1B | Q4_K_M | 22 GB | ~71 tok/s ModelFit est. |
| Poolside Laguna XS.233B | Q4_K_M | 25 GB | no estimate |
| Laguna XS 2.133B | Q4_K_M | 22 GB | ~332 tok/s ModelFit est. |
| Qwen3 32B32B | Q4_K_M | 22 GB | ~71 tok/s ModelFit est. |
| DeepSeek-R1 Distill Qwen 32B32B | Q4_K_M | 21 GB | ~72 tok/s ModelFit est. |
| Olmo 3.1 32B Instruct32B | Q4_K_M | 21 GB | ~73 tok/s ModelFit est. |
| Qwen3 30B30B | Q4_K_M | 24 GB | ~287 tok/s ModelFit est. |
| NVIDIA Nemotron Cascade 2 30B-A3B30B MoE | Q6_K | 26 GB | ~411 tok/s our est. |
| Cohere North Mini Code30B | Q4_K_M | 20 GB | no estimate |
| Nemotron 3.5 Lightning 30B-A3B30B MoE | Q4_K_M | 27 GB | ~250 tok/s ModelFit est. |
| GLM-4.7-Flash 30B-A3B30B MoE | Q4_K_M | 20 GB | ~273 tok/s ModelFit est. |
| Granite Switch 4.1 30B30B | Q4_K_M | 23 GB | ~67 tok/s ModelFit est. |
| Muse Glimmer 30B29.8B | Q4_K_M | 18.3 GB | ~90 tok/s ModelFit est. |
| Granite 4.2 30B29.3B | Q4_K_M | 17.9 GB | ~79 tok/s ModelFit est. |
| AREX-2 27B27.4B | Q4_K_M | 18.6 GB | ~87 tok/s ModelFit est. |
| Gemma 2 27B Instruct27B | Q4_K_M | 23 GB | ~80 tok/s ModelFit est. |
| Gemma 3 27B Instruct27B | Q4_K_M | 23 GB | ~84 tok/s ModelFit est. |
| Qwen3.5 27B Instruct27B | Q8_0 | 29 GB | ~36 tok/s our est. |
| Qwen3.6 27B27B | Q4_K_M | 19.4 GB | ~89 tok/s ModelFit est. |
| Jebadiah 27B27B | Q5_K_M | 21 GB | ~78 tok/s ModelFit est. |
| Mistral Small 3.124B | Q8_0 | 25 GB | ~58 tok/s ModelFit est. |
| LFM2 24B-A2B Instruct24B MoE | Q4_K_M | 15.3 GB | ~366 tok/s ModelFit est. |
| Devstral Small 2 24B24B | Q4_K_M | 16.6 GB | ~93 tok/s ModelFit est. |
| Mistral Small 22B22B | Q4_K_M | 18.4 GB | ~105 tok/s ModelFit est. |
| Qwen2.5 14B Instruct14B | Q4_K_M | 12.3 GB | ~142 tok/s ModelFit est. |
| Phi-3 Medium 14B14B | Q4_K_M | 12.3 GB | ~90 tok/s our est. |
| Phi-4 14B14B | Q8_0 | 15.9 GB | ~91 tok/s ModelFit est. |
| Qwen2.5 Coder 14B14B | Q4_K_M | 12.3 GB | ~142 tok/s ModelFit est. |
| Qwen3 14B14B | Q8_0 | 17.3 GB | ~91 tok/s ModelFit est. |
| DeepSeek-R1 Distill Qwen 14B14B | Q8_0 | 16.0 GB | ~92 tok/s ModelFit est. |
| Ministral 3 14B13.5B | Q4_K_M | 10.3 GB | ~142 tok/s ModelFit est. |
| Mellum2.1 12B-A2.5B Thinking12.1B MoE | Q4_K_M | 9.3 GB | ~589 tok/s our est. |
| Mistral Nemo 12B12B | Q8_0 | 13.4 GB | ~108 tok/s ModelFit est. |
| Gemma 3 12B Instruct12B | Q4_K_M | 10.7 GB | ~161 tok/s ModelFit est. |
| Gemma 4 12B12B | Q8_0 | 14.1 GB | ~106 tok/s ModelFit est. |
| GLM-4-9B-Chat9.4B | Q8_0 | 11.2 GB | ~140 tok/s ModelFit est. |
| Gemma 2 9B Instruct9B | Q4_K_M | 8.2 GB | ~173 tok/s ModelFit est. |
| Ornith 1.0 9B9B | Q4_K_M | 6.7 GB | ~225 tok/s ModelFit est. |
| MiMo-V2.6-Distill-Qwen-9B9B | Q4_K_M | 6.9 GB | ~219 tok/s ModelFit est. |
| Jebadiah 9B9B | Q8_0 | 11.0 GB | ~150 tok/s ModelFit est. |
| Granite 4.2 8B8.8B | Q4_K_M | 6.1 GB | ~204 tok/s ModelFit est. |
| Granite Switch 4.1 8B8.8B | Q4_K_M | 7.6 GB | ~175 tok/s ModelFit est. |
| MiniCPM-V 4.5 8B8.7B | Q4_K_M | 6.8 GB | ~219 tok/s ModelFit est. |
| Ministral 3 8B8.4B | Q4_K_M | 7.1 GB | ~194 tok/s ModelFit est. |
| LFM2.5 8B-A1B8.3B MoE | Q4_K_M | 6.6 GB | ~421 tok/s ModelFit est. |
| Llama 3.1 8B Instruct8B | Q8_0 | 9.2 GB | ~152 tok/s ModelFit est. |
| Qwen3 8B8B | Q8_0 | 9.3 GB | ~149 tok/s ModelFit est. |
| Granite 4.1 8B Instruct8B | Q4_K_M | 6.6 GB | ~198 tok/s ModelFit est. |
| Mistral 7B Instruct7B | Q8_0 | 8.4 GB | ~159 tok/s ModelFit est. |
| Qwen2.5 Coder 7B7B | Q4_K_M | 6.6 GB | ~240 tok/s ModelFit est. |
| DeepSeek-R1 Distill Qwen 7B7B | Q8_0 | 8.7 GB | ~164 tok/s ModelFit est. |
| Gemma 4 E4B4.5B | Q8_0 | 8.7 GB | ~225 tok/s ModelFit est. |
| Qwen3.5 4B Instruct4B | Q8_0 | 5.4 GB | ~239 tok/s ModelFit est. |
| NVIDIA Nemotron 3.5 Bolt E4B4B | Q4_K_M | 9.1 GB | ~267 tok/s ModelFit est. |
| Phi-3 Mini 3.8B3.8B | Q4_K_M | 4.3 GB | ~308 tok/s our est. |
| Phi-4 Mini 3.8B3.8B | Q8_0 | 4.9 GB | ~239 tok/s ModelFit est. |
| Granite 4.2 3B3.7B | Q4_K_M | 3.2 GB | ~351 tok/s ModelFit est. |
| Ministral 3 3B3.4B | Q4_K_M | 4.1 GB | ~291 tok/s ModelFit est. |
| Qwen2.5 3B Instruct3B | Q4_K_M | 3.6 GB | ~380 tok/s ModelFit est. |
| Llama 3.2 3B Instruct3B | Q4_K_M | 3.6 GB | ~346 tok/s ModelFit est. |
| Granite 4.1 3B Instruct3B | Q4_K_M | 3.0 GB | ~350 tok/s ModelFit est. |
| Gemma 4 E2B2.3B | Q8_0 | 5.7 GB | ~380 tok/s ModelFit est. |
| Gemma 2 2B Instruct2B | Q4_K_M | 2.8 GB | ~373 tok/s ModelFit est. |
| Qwen3.5 2B Instruct2B | Q4_K_M | 2.8 GB | ~382 tok/s ModelFit est. |
| LFM3 1.7B1.73B | Q4_K_M | 2.1 GB | ~484 tok/s ModelFit est. |
| Qwen2.5 1.5B Instruct1.5B | Q4_K_M | 2.5 GB | ~491 tok/s ModelFit est. |
| Llama 3.2 1B Instruct1B | Q4_K_M | 2.0 GB | ~516 tok/s ModelFit est. |
| Gemma 3 1B Instruct1B | Q8_0 | 2.0 GB | ~492 tok/s ModelFit est. |
| Qwen3.5 0.8B Instruct0.8B | Q4_K_M | 1.8 GB | ~529 tok/s ModelFit est. |
| Qwen2.5 0.5B Instruct0.5B | Q4_K_M | 1.8 GB | ~611 tok/s ModelFit est. |
| SmolLM2 360M0.36B | Q4_K_M | 1.5 GB | ~523 tok/s ModelFit est. |
Speeds come from ModelFit's estimates (CC BY 4.0) where it covers the model, otherwise from our bandwidth formula. Method
Fits only at 3-bit or lower
Too big for this machine
Mixtral 8x7B Instruct (32 GB), Llama 3.3 70B (44 GB), Llama 3.1 70B Instruct (44 GB), DeepSeek-R1 Distill Llama 70B (44 GB), Kolibri 1 (46 GB), Qwen3-Next 80B-A3B (53 GB), Nemotron 3 Super 120B-A12B (67 GB), gpt-oss-120b (68 GB), Llama 4 Scout (70 GB), Ling-3.0-flash (71 GB), Qwen3.5 122B-A10B Instruct (75 GB), Nemotron 3.5 Super VL 120B-A12B (76 GB) and more.
Questions
What is the best AI model for the RTX 5090 (32 GB)?
Qwen3.6-35B-A3B at Q4_K_M: the largest model on our shortlist of current models that runs at 4-bit or better at a usable speed (about 346 tok/s, ModelFit est.).
How much memory can a model use on the RTX 5090 (32 GB)?
About 31 GB of the 32 GB.
How fast is the RTX 5090 (32 GB) for local LLMs?
Decode speed is limited by memory bandwidth: 1792 GB/s. A model that reads 10 GB per token runs at under 179 tokens per second.