Self-Hosted AI: On-Premise, Private Cloud and Air-Gapped

Self-hosted AI means running the AI on infrastructure you control — your servers, your private cloud, or a network with no internet connection at all — instead of sending your data to somebody else's API.

For a growing set of organisations this stopped being a preference and became a requirement. Hospitals, banks, defence suppliers, public administrations and anyone bound by professional secrecy reach the same conclusion: the data cannot leave.

This guide covers what self-hosting actually involves, the three deployment models including air-gapped, what it costs compared to an API, where it genuinely wins and where it does not, and how to choose an on-premise AI platform without ending up maintaining a research project.

What Self-Hosted AI Actually Means

The term covers two separable things, and conflating them causes most of the confusion.

The model       →  an open-weight LLM running on your GPUs
The platform    →  retrieval, indexing, permissions, interface, monitoring

You can self-host both, or just the platform while calling an external model.
Only hosting both keeps every byte inside your perimeter.

Most organisations that say "we need self-hosted AI" actually need the second line: the documents and the retrieval must stay inside. Whether the model itself is local is a separate decision with different economics.

The Three Deployment Models

Private cloud

The platform runs in your own cloud tenancy — your account, your region, your network rules. You get most of the control with none of the hardware procurement. For a lot of organisations this is the pragmatic answer, and combined with a European provider it satisfies most data-residency requirements.

On-premise

The platform runs in your datacentre, on hardware you own. Full control, and full responsibility: you handle capacity, upgrades, backups and GPU availability. This is what regulated sectors usually mean when they say on-premise, and it is the hardest test a vendor can pass.

Air-gapped

No internet connection at all. Updates arrive on physical media, and the system has to work with no external call of any kind. Defence, critical infrastructure and classified environments operate this way.

Air-gapped deployment is the honest test of an architecture. A platform that quietly calls an external API for embeddings, telemetry or licence checks simply will not run — and you find out during installation rather than in the sales cycle.

Why Organisations Self-Host

Four reasons, in rough order of how often they are the deciding factor.

• Legal obligation. Professional secrecy, medical confidentiality, classified material, or sector rules that simply forbid the transfer.
• Extraterritorial exposure. Hosting in Europe is not sufficient if the provider is subject to non-European law — the point we develop in digital sovereignty.
• Cost at volume. Per-token pricing is excellent for prototypes and unforgiving at scale. Past a certain sustained load, owned hardware becomes cheaper.
• Independence. No deprecated model, changed pricing or altered terms can disrupt a system running on your own machines.

The Honest Cost Comparison

Vendors on both sides misrepresent this, so let us be precise about the shape of it rather than inventing numbers.

API           low fixed cost  +  cost per request
              →  cheap to start, scales linearly with usage forever

Self-hosted   high fixed cost  +  near-zero marginal cost
              →  expensive to start, then usage is essentially free

There is a crossover point, and where it falls depends entirely on your volume, your model choice and what your infrastructure already costs. The mistake is comparing an API bill against hardware alone: the real self-hosted cost includes GPUs, the people who operate them, and the electricity.

The pragmatic reading: if usage is low or unpredictable, an API is cheaper and you should not pretend otherwise. If load is high and steady, self-hosting wins on cost as well as on control. And if the data legally cannot leave, cost is not the deciding variable at all.

What You Take On

Being straight about the downsides, because they are real and vendors skip them.

• Hardware. Running a capable model needs GPUs, and GPU capacity is a procurement problem, not a purchase.
• Operations. Someone monitors, patches and upgrades this. If nobody owns it, it rots.
• Model choice. You pick from open-weight models. The best of them are strong, but you are not getting the latest frontier model the week it ships.
• Upgrade discipline. Nobody updates it for you. That is the point, and it is also work.

The failure mode is predictable: a proof of concept assembled from open-source components by one enthusiastic engineer, which works beautifully until that person changes team. Self-hosting is an operational commitment, not a weekend project.

Self-Hosting Without Building It Yourself

There is a middle path that gets overlooked. You do not have to choose between an external API and assembling your own stack from parts.

A packaged platform deployed on your infrastructure gives you the data guarantees of self-hosting with a supported product rather than a pile of components you maintain. You own the deployment; someone else owns the roadmap, the security patches and the upgrade path.

For most organisations this is the right trade. Assembling retrieval, indexing, permissions, monitoring and an interface from open-source parts is entirely possible and consistently underestimated — as we discuss in our API guide, the prototype is an afternoon and the production system is months.

How to Evaluate an On-Premise AI Platform

1. Will it run air-gapped? Even if you do not need it, the answer reveals whether the architecture has hidden external dependencies.
2. What exactly leaves the network? Telemetry, licence checks, model calls. Get it in writing.
3. Which models can it use? Open-weight models locally, external APIs, or both — and can you switch without rebuilding?
4. What hardware does it actually need? Ask for the specification for your expected load, not a minimum.
5. How do upgrades work offline? A platform that assumes a package repository is not really on-premise ready.
6. Are permissions enforced at retrieval? Self-hosting does not fix access control; that is a separate property.

Self-Hosted AI with QAnswer

QAnswer is built to be deployed rather than only consumed. The same platform runs in a private cloud, on-premise, or air-gapped, and the deployment model does not change what it can do.

• Your documents never leave. Indexing and retrieval happen inside your perimeter.
• Model flexibility. Run open-weight models locally, or connect an external provider where policy allows.
• Permissions at retrieval, so an answer never exposes a document the user could not open — see governance.
• A citation on every answer, which is what makes an internal assistant auditable.
• ISO 27001 and ISO 42001 certified — security and AI governance.

It is deployed at the European Parliament, the European Commission, the Smithsonian Institution and Rockwell Automation. Connect it to SharePoint, Confluence, databases or document stores, or embed it through the APIs.

Frequently Asked Questions

What is self-hosted AI?

AI running on infrastructure you control rather than a third-party API. It can mean hosting the model, the surrounding platform, or both — and only hosting both keeps all data inside your perimeter.

What is the difference between self hosted AI and on-premise AI?

They overlap. On-premise specifically means your own datacentre. Self-hosted is broader and includes running in your own private cloud tenancy, which most organisations find more practical.

Is self-hosted AI cheaper?

At high, steady volume, usually yes, because the marginal cost per request approaches zero. At low or spiky volume, no — an API is cheaper and simpler. The crossover depends on your load and hardware costs.

What hardware do I need?

It depends on the model and the number of concurrent users, so treat any single answer with suspicion. Ask a vendor to size it for your expected load rather than quoting a minimum specification.

Can it work with no internet connection?

A properly designed platform can run fully air-gapped. Many cannot, because they call external services for embeddings, telemetry or licensing. Ask specifically.

Are self-hosted models less capable?

Open-weight models trail the frontier, but for retrieval-based tasks the gap matters much less than people assume — answer quality is driven mainly by retrieval relevance, as we set out in how accurate is ChatGPT.

Do I need a team to run it?

You need someone who owns it. Building your own stack requires real engineering capacity; deploying a packaged platform on your infrastructure requires ordinary operations capability.

The Bottom Line

Self-hosted AI is the answer when the data cannot leave — and for a lot of organisations that is not negotiable, so the cost debate is beside the point.

The mistake is treating it as a binary between an external API and building everything yourself. Deploying a supported platform on your own infrastructure gives you the guarantees without the research project, and that is the option most organisations should be evaluating.

‍

Deploy AI on your own infrastructure — private cloud, on-premise or air-gapped — with QAnswer. Explore AI Assistants and data sovereignty.

Learn more at www.qanswer.ai

Discussing a self-hosted deployment? Contact us or email info@the-qa-company.com