When Should AI Queries Run Locally?
How to measure which AI queries can run locally: routing architecture, cost, latency and confidentiality.
The essentials in 5 points
- The share of queries that can run locally must be measured on your real traffic; there is no reliable universal percentage.
- The architecture that follows is the local-first router: local by default, cloud switch on a difficulty threshold.
- A query served locally has a near-zero marginal cost: the cloud bill narrows to the remaining complex fraction.
- The gain is not only financial: lower latency and data that never leaves the device, a strong compliance argument.
- Before sizing, audit your real traffic: that is the measurement that tells you what share fits locally.
The short answer
Many everyday AI queries are simple: rephrase text, extract information, classify a message, summarize a document, or answer a factual question. But the proportion that a local model can handle depends on context, hardware, and the required quality level. The practical consequence is to measure that proportion on a representative sample before sizing the architecture.
The architecture that exploits this finding is the local-first router. Each query goes to a local model first, and only switches to the cloud if a difficulty threshold is crossed. The rest of this article details how that switch works, what it changes for cost, latency and confidentiality, how to audit your traffic to size it, and where the honest limits are.
The figure and where it comes from
The important point is not a generic figure but a reversal of the burden of proof: the question is no longer “what can run locally?” but “what really needs the cloud?”. The answer comes from a real query set, explicit quality criteria, and a measurable escalation threshold.
That reversal holds because small models have caught up. By the summer of 2026, modestly sized open source models match far larger ones on everyday tasks, and a nine-billion-parameter model is enough to cover a wide share of daily work. The share reserved for the cloud corresponds to queries that demand long reasoning, very specialized knowledge, or a creativity that only the high end still reaches.
The local-first router architecture
The principle is simple. A router receives each query and directs it to a local model first. A difficulty-estimation mechanism decides whether the local answer is enough or whether the query should escalate to a more capable cloud model. The threshold can be tuned on various signals: length and complexity of the request, the local model’s confidence, the presence of sensitive data that must not be exposed.
The appeal of this design is that it flips the default. A cloud-by-default architecture sends everything to the provider, including trivial queries that do not need it. A local-first architecture only calls the cloud for the fraction that justifies it. The result can cost less, expose less data, and reduce network dependency, provided that quality and operating costs are measured.
What it changes: cost, latency, confidentiality
Cost. A query served locally has a near-zero marginal cost. Once the hardware is amortized, there is no per-token billing for the local share of traffic. If most queries go local, the cloud bill shrinks to the remaining complex fraction. For an organization handling a large volume of everyday queries, the cumulative gap over a year is significant, without a single price tag capturing it: it all depends on the traffic profile.
Latency. A local model does not cross the network. No round trip to a distant data center, no dependence on connection quality. For interactive uses, code autocompletion or continuous assistance, this responsiveness changes the experience.
Confidentiality. This is the argument that goes beyond cost. A query served locally never lets the data leave the device. Recent disputes around data sharing by online AI services are a reminder that as soon as data leaves the device, it escapes control and can leak through unforeseen channels. For a regulated profession, for medical or financial data, local-first is not an optimization, it is the guarantee that the data never goes to a third party.
How to audit your traffic to size it
You do not size a local-first architecture by guesswork. The right method starts with an audit of real AI traffic. Log the queries sent to the assistants in place, classify them by complexity and by data sensitivity, then estimate the share that would fit on a local model. This upfront measurement answers two questions: what hardware to plan for the local load, and what fraction of traffic will legitimately stay in the cloud.
Without this audit, two mistakes loom. Over-sizing the local infrastructure for traffic that does not justify it, or conversely underestimating the complex share and degrading quality by forcing local where the cloud is called for. The value of a local-first router lies in that tuning, and the tuning lies in the measurement.
The honest limits
The approach has a cost worth naming. First, maintenance: a fleet of local models has to be updated, monitored, corrected. That load does not exist with a fully managed cloud API. Then, quality on complex cases: the 30% of queries that justify the cloud switch really do justify it, and a poorly tuned router that keeps them local produces subpar answers. Finally, MLOps: operating a router, measuring difficulty, handling switches and monitoring quality demands skills that not every team has in house.
These limits do not undermine the thesis, they frame it. Local-first cuts the bill and the leak risk, it does not remove the need for know-how to run the whole thing. It is an architectural trade-off, not a turnkey solution.
Where to start
The first step is not to pick a model, it is to measure. An audit of AI traffic establishes the share truly servable locally, quantifies the possible saving, and identifies the data that must never leave. From there, the architecture is sized on facts, not on intuition.
Colombani.ai designs this kind of sovereign AI architecture (local-first router, controlled cloud switch, data kept in Europe) and measures the expected return through a ROI assessment that quantifies the saving and the load before any investment. The expertise involved is certified by Anthropic (Claude Certified Architect), and the local AI training then lets your teams run the setup autonomously.