The essentials in 5 points
- The constraint that decides everything is not the GPU, it is the available memory and its bandwidth.
- Quantization makes a model fit in four times less memory, with a quality loss that is almost imperceptible at 4-bit and 8-bit.
- Each memory tier has its realistic use: 8 GB for code assistance and extraction, 48 GB and up for a feel close to the proprietary high end.
- “It fits” and “it is fluid” are two different questions: capacity gets the model in, bandwidth makes it pleasant to use.
- The open source ecosystem (Qwen, GLM, Kimi, Mistral, Gemma, DeepSeek, Llama) is now on par with proprietary models for a wide share of uses.
The short answer
The right model is not the biggest one you can download, it is the biggest one that fits comfortably in your memory while leaving room for context. To estimate what fits, one simple rule is enough: a model quantized to 4-bit takes about half a byte per parameter. A 7-billion-parameter model therefore weighs roughly 4 GB, plus context and system overhead. That is why an 8 GB machine already runs a useful code assistant, while a 48 GB machine reaches a quality that practitioners describe as close to the proprietary high end.
The rest of this article details the memory tiers, explains quantization without jargon, and separates two questions people often confuse: does the model fit, and is it fluid.
Why RAM decides everything
A language model is a big bundle of numbers, its parameters. To generate text, the machine has to read those numbers at each step. Two hardware properties matter here, and they are not the same thing.
The first is capacity: the model has to fit in memory, otherwise it will not load, or it pages from disk and becomes very slow. The second is memory bandwidth: the speed at which the machine reads those numbers, which sets how many words are generated per second. A Mac with unified memory owes its reputation in local inference to this high bandwidth, not to a dedicated GPU.
The GPU, as it happens, helps but is not essential. 1-bit inference projects let very large models run on a plain processor, and the Apple Silicon ecosystem has made the Mac a credible inference platform. So the real sizing question remains memory, not the graphics card.
Quantization, explained simply
By default, each model parameter is stored at 16-bit precision, or 2 bytes. Multiply by several billion parameters and the memory footprint quickly moves out of a workstation’s reach.
Quantization stores those same parameters with fewer bits. At 8-bit, you halve the footprint ; at 4-bit, you quarter it. A model that demanded 28 GB at full precision then fits in roughly 7 GB. You lose a bit of fineness in the numbers, but for most business tasks the difference is almost imperceptible down to 4-bit. Below that, quality degrades noticeably, except for architectures designed for low precision.
The practical implication: never reason about a model’s raw size, reason about its quantized size. That is what decides what your machine can host.
The guide by memory tier
8 GB: targeted assistance. A model of 7 to 9 billion parameters at 4-bit fits and leaves room for context. Realistic uses are code autocompletion, extracting information from a document, classification, summarization and light tool-calling. The small Qwen, Gemma and Llama models are cut for this tier, and some rival far larger models on targeted tasks.
16 GB: the generalist assistant. You move up to 13 to 27 billion parameters quantized. Reasoning becomes solid, local RAG over your documents works well, and the assistant holds a coherent working conversation. This is the comfortable entry tier for daily use.
32 GB: serious coding and reasoning. Models of 30 to 70 billion parameters quantized become accessible. This is the tier where agentic coding, reliable tool-calling and multi-step reasoning hold up. The Kimi (agentic coding), DeepSeek (reasoning) and Mistral families find their ground here.
48 GB and up: the high-end feel. At this level, a large model quantized to 8-bit reaches a quality that practitioners describe as close to the best proprietary offering, tool-calling included. The GLM and Qwen families in their large variants target this tier. It is the threshold of the sovereign workstation, able to handle sensitive files without ever calling the cloud.
”It fits” is not “it is fluid”
This is the most frequent confusion. A 35-billion-parameter model loads on a 16 GB machine by paging from disk: it fits, but every word makes you wait. Conversely, the same model on a high-bandwidth machine generates at a comfortable speed.
The sizing lesson: do not aim for the capacity limit. Keep headroom for context and for generation comfort. A model one notch below the theoretical maximum, but fluid, serves better day to day than a larger model that crawls. For a workstation, fluidity is the difference between a tool you use and a tool you abandon.
The enterprise use case: the sovereign workstation
The 48 GB tier opens a scenario the cloud cannot offer: a machine where data never leaves the device. For a regulated profession, a law firm, a healthcare or finance player, this is an argument that goes beyond cost. Recent disputes around data sharing by online AI services are a reminder that as soon as data leaves the device, it escapes control and can leak through unforeseen channels.
A sovereign workstation answers that requirement: the model runs locally, sensitive documents stay on the device, and GDPR compliance is reasoned at the scale of a machine rather than a third-party provider. The cloud frontier model stays available for the small fraction of queries that justify it, but it is no longer the mandatory route for all the work.
Where to start
The right starting point is not to download the biggest model possible, but to start from your hardware and your uses. Measure the memory actually available on the relevant machines, identify the tasks to cover, then choose the most capable quantized model that stays fluid on that hardware.
Colombani.ai trains technical teams in this approach through the Local AI, your data stays with you course (model choice by hardware constraint, quantization, local inference environment), Qualiopi-certified and eligible for OPCO funding. To size a fleet of sovereign workstations, Colombani.ai designs sovereign AI architectures that keep your data in Europe and on your machines. The expertise involved is certified by Anthropic (Claude Certified Architect).
Sources
Have a project in mind?
Describe your situation. Straight answer within 48 hours.