Which AI Model for Your RAM: A Guide
Choosing an open source AI model by your available memory: what runs on 8, 16, 32 or 48 GB, quantization made simple, and a workstation that keeps your data on site.
The essentials in 5 points
- The main criterion is not the GPU, but the available memory and its bandwidth.
- Quantization makes a model fit in four times less memory, with a quality loss that is almost imperceptible at 4-bit and 8-bit.
- Each memory tier has its realistic use: 8 GB for code assistance and extraction, 48 GB and up for a quality close to the best proprietary models.
- “It fits” and “it is fluid” are two different questions: capacity gets the model in, bandwidth makes it pleasant to use.
- The open source ecosystem (Qwen, GLM, Kimi, Mistral, Gemma, DeepSeek, Llama) is now on par with proprietary models for a wide share of uses.
The short answer
The right model is not the biggest one you can download, it is the biggest one that fits comfortably in your memory while leaving room for context. To estimate what fits, one simple rule is enough: a model quantized to 4-bit takes about half a byte per parameter.
A 7-billion-parameter model therefore weighs roughly 4 GB, plus context and system overhead. That is why an 8 GB machine already runs a useful code assistant, while a 48 GB machine runs models ten times larger.
For a mixture-of-experts (MoE) model, memory must hold every parameter, but only a fraction is used for each generated token, which can improve fluidity.
The rest of this article details the memory tiers, explains quantization without jargon, and separates two questions people often confuse: does the model fit, and is it fluid.
Want to run a local model for your team? Colombani.ai’s local AI training covers choosing a model for your available memory, installing it and using it on your documents, with no data sent to the cloud.
Why RAM decides everything
A language model is a very large set of numbers, called parameters. To generate text, the machine has to read those numbers at each step. Two hardware properties matter here, and they are not the same thing.
The first is capacity: the model has to fit in memory, otherwise it will not load, or it pages from disk and becomes very slow. The second is memory bandwidth: the speed at which the machine reads those numbers, which sets how many words are generated per second. A Mac with unified memory owes its reputation in local inference to this high bandwidth, not to a dedicated GPU.
The GPU, as it happens, helps but is not essential. 1-bit inference projects let very large models run on a plain processor, and the Apple Silicon ecosystem has made the Mac a credible inference platform. So the real sizing question remains memory, not the graphics card.
Quantization, explained simply
By default, each model parameter is stored at 16-bit precision, or 2 bytes. Multiply by several billion parameters and the memory footprint quickly moves out of a workstation’s reach.
Quantization stores those same parameters with fewer bits. At 8-bit, you halve the footprint ; at 4-bit, you quarter it. A model that demanded 28 GB at full precision then fits in roughly 7 GB. You lose a bit of fineness in the numbers, but for most business tasks the difference is almost imperceptible down to 4-bit. Below that, quality degrades noticeably, except for architectures designed for low precision.
The practical implication: never reason about a model’s raw size, reason about its quantized size. That is what decides what your machine can host.
The guide by memory tier
8 GB: targeted assistance. A model of 7 to 9 billion parameters at 4-bit fits and leaves room for context. Realistic uses are code autocompletion, extracting information from a document, classification, summarization and light tool-calling. The small Qwen, Gemma and Llama models are cut for this tier, and some rival far larger models on targeted tasks.
16 GB: the generalist assistant. You move up to 13 to 27 billion parameters quantized. Reasoning becomes solid, local RAG over your documents works well, and the assistant holds a coherent working conversation. This is the comfortable entry tier for daily use.
32 GB: serious coding and reasoning. Models of 30 to 45 billion parameters quantized to 4-bit become accessible; a 70-billion model, at 35 GB of weights before context, overflows. This is the tier where agentic coding, reliable tool-calling and multi-step reasoning hold up. The Kimi (agentic coding), DeepSeek (reasoning) and Mistral families are suited to this tier.
48 GB and up: quality close to the best proprietary models. At this level, a large model quantized to 8-bit reaches a quality that practitioners consider close to the best proprietary models, tool-calling included. The GLM and Qwen families in their large variants target this tier. It is the threshold of the sovereign workstation, able to handle sensitive files without ever calling the cloud.
”It fits” is not “it is fluid”
This is the most frequent confusion. A 35-billion-parameter model loads on a 16 GB machine by paging from disk: it overflows physical memory, the system fetches the rest from disk, and every word makes you wait. Conversely, the same model on a high-bandwidth machine generates at a comfortable speed.
The sizing lesson: do not aim for the capacity limit. Keep headroom for context and for generation comfort. A model one notch below the theoretical maximum, but fluid, serves better day to day than a larger model that crawls. On a workstation, a slow tool is soon abandoned.
Mixture-of-experts models: large in memory, lighter to run
Some models split their parameters into several groups called experts. For each generated token, a selection of those groups is used, together with the shared parts of the model. This architecture, called mixture of experts or MoE, reduces the work needed at each step.
For example, Qwen3-30B-A3B, announced in April 2025 under the Apache 2.0 licence, has 30.5 billion parameters, of which 3.3 billion are active per generated token, spread across 128 experts with 8 used at a time. Its quantized Q4_K_M file weighs about 18.6 GB. You still need memory for context, software and the system.
To size your workstation, look at two figures: total parameters to estimate the memory needed, active parameters to gauge the generation work. To avoid on-demand loading, every expert must stay in memory, including those momentarily unused.
A mixture-of-experts model can therefore generate faster than a dense model of comparable total size. Speed still depends on hardware, software and context: the ratio between total and active parameters does not translate into an equivalent speed-up, it has to be measured. On a 32 GB workstation, this example deserves a trial with your own documents and tasks.
The enterprise use case: the sovereign workstation
The 48 GB tier makes possible what the cloud cannot offer: a machine where data never leaves the device. For a regulated profession, a law firm, a healthcare or finance player, this argument outweighs cost. Recent disputes around data sharing by online AI services are a reminder that as soon as data leaves the device, it escapes control and can leak through unforeseen channels.
A sovereign workstation answers that requirement: the model runs locally, sensitive documents stay on the device, and GDPR compliance is assessed at the scale of a machine rather than a third-party provider. The cloud frontier model stays available for the small fraction of queries that justify it, but it is no longer the mandatory route for all the work.
For an organisation that wants to host its models on its own machines or with a European provider, Colombani.ai’s sovereign architecture offer sets the sizing and security.
Where to start
Rather than downloading the biggest model possible, start from your hardware and your uses. Measure the memory actually available on the relevant machines, identify the tasks to cover, then choose the most capable quantized model that stays fluid on that hardware.
Colombani.ai trains technical teams in this approach through the Local AI, your data stays with you course (model choice by hardware constraint, quantization, local inference environment), delivered by a Qualiopi-certified provider and eligible for OPCO funding. To size a fleet of sovereign workstations, Colombani.ai designs sovereign AI architectures that keep your data in Europe and on your machines. The expertise involved is certified by Anthropic (Claude Certified Architect).
Sources
Frequently asked questions
What size AI model can I run with 8 GB of RAM?
With 8 GB, an open source model of 7 to 9 billion parameters quantized to 4-bit fits in memory and leaves room for context. That is enough for code autocompletion, information extraction, classification, summarization and light tool-calling. The Qwen, Gemma and Llama families offer models of this size designed for constrained hardware.
What is model quantization?
Quantization reduces the precision with which each model parameter is stored. At 16-bit, a parameter takes 2 bytes ; at 4-bit, about half a byte, four times less memory. You lose a little precision, but it stays almost imperceptible at 8-bit and 4-bit for most tasks. This is what lets a large model fit on consumer hardware.
Do I need a GPU to run an AI model locally?
No, not necessarily. A Mac with unified memory runs quantized models very well thanks to its high memory bandwidth. On a PC, a GPU speeds up inference a lot, but 1-bit inference projects also let large models run on a plain processor. The real question is not the GPU but the available memory and its bandwidth.
What is the difference between a model that fits and a model that is fluid?
A model fits if its size fits within the available memory : that is a matter of capacity. It is fluid if it generates text at a comfortable speed, which depends on memory bandwidth. A machine can make a large model fit by paging from disk, but generation will be slow. For daily use, aim for fluidity, not just capacity.
Does a local open source model match proprietary models?
For many common business uses, yes. By the summer of 2026, several open source families reach frontier level on standard benchmarks, and practitioners describe a quality close to the best proprietary models on a 48 GB machine. The cloud frontier model remains useful for the rare, most complex queries, not for the whole workload.
Does a mixture-of-experts (MoE) model need less memory?
At equal total parameters and quantization, the weights take a comparable amount of space: every expert must be kept for fully in-memory execution. Only a fraction is used for each generated token, which can speed up the response. Qwen3-30B-A3B has 30.5 billion parameters, of which 3.3 billion are active. Its Q4_K_M file weighs about 18.6 GB, plus context, software and the system. Actual speed depends on the hardware and software used.