Graphics cards
Which AI models run on the RTX 4090 (24 GB)?
The RTX 4090 (24 GB) has 24 GB of memory, about 23 GB of it usable for a model, and 1008 GB/s of memory bandwidth. 80 of the 110 models in our catalog run on it at 4-bit or better.
Data as of 2 November 2026 · Vendor specs
24 GBmemory
~23 GB usable for a model
1008 GB/sbandwidth, sets the speed
80models run at 4-bit or better
Models that run on it
| Model | Best build | Memory | Speed |
|---|---|---|---|
| Gemma 4 31B30.7B | Q4_K_M | 21 GB | ~41 tok/s ModelFit est. |
| Qwen3-Coder-30B-A3B30.5B MoE | Q4_K_M | 20 GB | ~189 tok/s ModelFit est. |
| Nemotron 3 Nano 30B-A3B30B MoE | Q4_K_M | 19.6 GB | ~305 tok/s our est. |
| Qwen3.8-27B27.3B | Q4_K_M | 17.9 GB | ~51 tok/s ModelFit est. |
| Gemma 4 26B-A4B25.2B MoE | Q4_K_M | 17.4 GB | ~168 tok/s ModelFit est. |
| gpt-oss-20b21B MoE | MXFP4 | 15.1 GB | ~169 tok/s ModelFit est. |
| Nemotron Nano 9B v29B | FP16 | 19.4 GB | ~31 tok/s our est. |
| Qwen3.5 9B Instruct9B | Q8_0 | 12.0 GB | ~92 tok/s ModelFit est. |
| Qwen2.5 7B Instruct7B | Q4_K_M | 6.6 GB | ~153 tok/s ModelFit est. |
| Gemma 3 4B Instruct4B | Q8_0 | 5.0 GB | ~160 tok/s ModelFit est. |
| Qwen3.5 35B-A3B Instruct35B MoE | Q4_K_M | 21 GB | ~242 tok/s ModelFit est. |
| Ornith 1.0 35B35B | Q4_K_M | 23 GB | ~242 tok/s ModelFit est. |
| Nanbeige4.3 33B33.1B | Q4_K_M | 22 GB | no estimate |
| Laguna XS 2.133B | Q4_K_M | 22 GB | ~226 tok/s ModelFit est. |
| Qwen3 32B32B | Q4_K_M | 22 GB | ~40 tok/s ModelFit est. |
| DeepSeek-R1 Distill Qwen 32B32B | Q4_K_M | 21 GB | ~41 tok/s ModelFit est. |
| Olmo 3.1 32B Instruct32B | Q4_K_M | 21 GB | ~41 tok/s ModelFit est. |
| Cohere North Mini Code30B | Q4_K_M | 20 GB | no estimate |
| Nemotron 3.5 Lightning 30B-A3B30B MoE | IQ4_XS | 20 GB | ~293 tok/s our est. |
| GLM-4.7-Flash 30B-A3B30B MoE | Q4_K_M | 20 GB | ~178 tok/s ModelFit est. |
| Granite Switch 4.1 30B30B | Q4_K_M | 23 GB | no estimate |
| Muse Glimmer 30B29.8B | Q4_K_M | 18.3 GB | ~51 tok/s ModelFit est. |
| Granite 4.2 30B29.3B | Q4_K_M | 17.9 GB | ~45 tok/s ModelFit est. |
| AREX-2 27B27.4B | Q4_K_M | 18.6 GB | ~50 tok/s ModelFit est. |
| Gemma 2 27B Instruct27B | Q4_K_M | 23 GB | ~46 tok/s ModelFit est. |
| Gemma 3 27B Instruct27B | Q4_K_M | 23 GB | ~48 tok/s ModelFit est. |
| Qwen3.5 27B Instruct27B | Q4_K_M | 17.4 GB | ~52 tok/s ModelFit est. |
| Qwen3.6 27B27B | Q4_K_M | 19.4 GB | ~51 tok/s ModelFit est. |
| Jebadiah 27B27B | Q5_K_M | 21 GB | ~44 tok/s ModelFit est. |
| Mistral Small 3.124B | Q4_K_M | 16.4 GB | ~37 tok/s our est. |
| LFM2 24B-A2B Instruct24B MoE | Q4_K_M | 15.3 GB | ~257 tok/s ModelFit est. |
| Devstral Small 2 24B24B | Q4_K_M | 16.6 GB | ~53 tok/s ModelFit est. |
| Mistral Small 22B22B | Q4_K_M | 18.4 GB | ~61 tok/s ModelFit est. |
| Qwen2.5 14B Instruct14B | Q4_K_M | 12.3 GB | ~84 tok/s ModelFit est. |
| Phi-3 Medium 14B14B | Q4_K_M | 12.3 GB | ~50 tok/s our est. |
| Phi-4 14B14B | Q8_0 | 15.9 GB | ~52 tok/s ModelFit est. |
| Qwen2.5 Coder 14B14B | Q4_K_M | 12.3 GB | ~84 tok/s ModelFit est. |
| Qwen3 14B14B | Q8_0 | 17.3 GB | ~52 tok/s ModelFit est. |
| DeepSeek-R1 Distill Qwen 14B14B | Q8_0 | 16.0 GB | ~52 tok/s ModelFit est. |
| Ministral 3 14B13.5B | Q4_K_M | 10.3 GB | ~84 tok/s ModelFit est. |
| Mellum2.1 12B-A2.5B Thinking12.1B MoE | Q4_K_M | 9.3 GB | ~331 tok/s our est. |
| Mistral Nemo 12B12B | Q8_0 | 13.4 GB | ~62 tok/s ModelFit est. |
| Gemma 3 12B Instruct12B | Q4_K_M | 10.7 GB | ~97 tok/s ModelFit est. |
| Gemma 4 12B12B | Q8_0 | 14.1 GB | ~61 tok/s ModelFit est. |
| GLM-4-9B-Chat9.4B | Q8_0 | 11.2 GB | ~83 tok/s ModelFit est. |
| Gemma 2 9B Instruct9B | Q4_K_M | 8.2 GB | ~105 tok/s ModelFit est. |
| Ornith 1.0 9B9B | Q4_K_M | 6.7 GB | ~141 tok/s ModelFit est. |
| MiMo-V2.6-Distill-Qwen-9B9B | Q4_K_M | 6.9 GB | ~137 tok/s ModelFit est. |
| Jebadiah 9B9B | Q8_0 | 11.0 GB | ~89 tok/s ModelFit est. |
| Granite 4.2 8B8.8B | Q4_K_M | 6.1 GB | ~127 tok/s ModelFit est. |
| Granite Switch 4.1 8B8.8B | Q4_K_M | 7.6 GB | ~106 tok/s ModelFit est. |
| MiniCPM-V 4.5 8B8.7B | Q4_K_M | 6.8 GB | ~137 tok/s ModelFit est. |
| Ministral 3 8B8.4B | Q4_K_M | 7.1 GB | ~119 tok/s ModelFit est. |
| LFM2.5 8B-A1B8.3B MoE | Q4_K_M | 6.6 GB | ~309 tok/s ModelFit est. |
| Llama 3.1 8B Instruct8B | Q8_0 | 9.2 GB | ~91 tok/s ModelFit est. |
| Qwen3 8B8B | Q8_0 | 9.3 GB | ~89 tok/s ModelFit est. |
| Granite 4.1 8B Instruct8B | Q4_K_M | 6.6 GB | ~122 tok/s ModelFit est. |
| Mistral 7B Instruct7B | Q8_0 | 8.4 GB | ~95 tok/s ModelFit est. |
| Qwen2.5 Coder 7B7B | Q4_K_M | 6.6 GB | ~153 tok/s ModelFit est. |
| DeepSeek-R1 Distill Qwen 7B7B | Q8_0 | 8.7 GB | ~99 tok/s ModelFit est. |
| Gemma 4 E4B4.5B | Q8_0 | 8.7 GB | ~142 tok/s ModelFit est. |
| Qwen3.5 4B Instruct4B | Q8_0 | 5.4 GB | ~152 tok/s ModelFit est. |
| NVIDIA Nemotron 3.5 Bolt E4B4B | Q4_K_M | 9.1 GB | ~173 tok/s ModelFit est. |
| Phi-3 Mini 3.8B3.8B | Q4_K_M | 4.3 GB | ~173 tok/s our est. |
| Phi-4 Mini 3.8B3.8B | Q8_0 | 4.9 GB | ~152 tok/s ModelFit est. |
| Granite 4.2 3B3.7B | Q4_K_M | 3.2 GB | ~243 tok/s ModelFit est. |
| Ministral 3 3B3.4B | Q4_K_M | 4.1 GB | ~192 tok/s ModelFit est. |
| Qwen2.5 3B Instruct3B | Q4_K_M | 3.6 GB | ~269 tok/s ModelFit est. |
| Llama 3.2 3B Instruct3B | Q4_K_M | 3.6 GB | ~239 tok/s ModelFit est. |
| Granite 4.1 3B Instruct3B | Q4_K_M | 3.0 GB | ~243 tok/s ModelFit est. |
| Gemma 4 E2B2.3B | Q8_0 | 5.7 GB | ~269 tok/s ModelFit est. |
| Gemma 2 2B Instruct2B | Q4_K_M | 2.8 GB | ~263 tok/s ModelFit est. |
| Qwen3.5 2B Instruct2B | Q4_K_M | 2.8 GB | ~271 tok/s ModelFit est. |
| LFM3 1.7B1.73B | Q4_K_M | 2.1 GB | ~375 tok/s ModelFit est. |
| Qwen2.5 1.5B Instruct1.5B | Q4_K_M | 2.5 GB | ~383 tok/s ModelFit est. |
| Llama 3.2 1B Instruct1B | Q4_K_M | 2.0 GB | ~411 tok/s ModelFit est. |
| Gemma 3 1B Instruct1B | Q8_0 | 2.0 GB | ~384 tok/s ModelFit est. |
| Qwen3.5 0.8B Instruct0.8B | Q4_K_M | 1.8 GB | ~428 tok/s ModelFit est. |
| Qwen2.5 0.5B Instruct0.5B | Q4_K_M | 1.8 GB | ~534 tok/s ModelFit est. |
| SmolLM2 360M0.36B | Q4_K_M | 1.5 GB | ~420 tok/s ModelFit est. |
Speeds come from ModelFit's estimates (CC BY 4.0) where it covers the model, otherwise from our bandwidth formula. Method
Fits only at 3-bit or lower
Too big for this machine
Sage-A1.5 (23 GB), Qwen3.6-35B-A3B (24 GB), Qwen3 30B (24 GB), Flint 40B-A2.5B (24 GB), Poolside Laguna XS.2 (25 GB), NVIDIA Nemotron Cascade 2 30B-A3B (26 GB), Mixtral 8x7B Instruct (32 GB), Llama 3.3 70B (44 GB), Llama 3.1 70B Instruct (44 GB), DeepSeek-R1 Distill Llama 70B (44 GB), Kolibri 1 (46 GB), Qwen3-Next 80B-A3B (53 GB) and more.
Questions
What is the best AI model for the RTX 4090 (24 GB)?
Gemma 4 31B at Q4_K_M: the largest model on our shortlist of current models that runs at 4-bit or better at a usable speed (about 41 tok/s, ModelFit est.).
How much memory can a model use on the RTX 4090 (24 GB)?
About 23 GB of the 24 GB.
How fast is the RTX 4090 (24 GB) for local LLMs?
Decode speed is limited by memory bandwidth: 1008 GB/s. A model that reads 10 GB per token runs at under 101 tokens per second.