Self-hosted AI means your organisation runs the model-serving infrastructure, rather than sending requests to a provider's model API. That can be an open-weight model on a GPU server in a Swiss data centre, or a Kubernetes inference platform inside your own private cloud.
For teams handling sensitive data, self-hosting can seem like the obvious choice. But confidentiality alone does not tell you whether to buy GPUs, use a managed regional service or combine both. The decision depends on the control you need, the quality your workload requires and what your team can reliably operate.
What self-hosting actually means
Private AI can describe several different arrangements. Before choosing an architecture or signing a contract, establish who runs the model, where it processes data and which responsibilities remain with your team.
| Approach | Who runs inference | Residency and privacy control | Engineering burden | Typical fit |
|---|---|---|---|---|
| Hosted model API | Model vendor | Contractual and endpoint-dependent | Low | Prototypes, variable demand, frontier capability |
| Managed AI platform | Cloud provider | Medium to high, depending on deployment type | Low to medium | Enterprise apps needing cloud IAM, networking and governance |
| VPC or private endpoint | Provider, usually on provider infrastructure | Stronger network isolation; residency is a separate question | Medium | Organisations that forbid public-internet AI traffic |
| Private-cloud AI | You, or a managed partner | High, if configured and governed well | High | Stable enterprise workloads with specific control requirements |
| Hybrid | Both, by policy | Depends on how routing is designed | High | Mixed sensitivity and mixed capability needs |
| On-premises | You | High, if configured and governed well | Very high | Regulated or offline environments, stable high utilisation |
| Edge | You, or the device operator | High for data kept on the device | Medium to high | Very low latency, or disconnected operation |
The categories overlap. Running an open-weight model on rented cloud GPUs inside your own VPC is simultaneously cloud AI and self-hosted AI. What matters is not the label but the answers to four separate questions: where the data travels, who operates inference, who is accountable for security, and which legal obligations attach to the use case.
A private endpoint is not private hosting
This is the distinction that most often gets collapsed, and it is expensive to get wrong late.
AWS documents PrivateLink as a way to reach Amazon Bedrock from a VPC without an internet gateway, so that the service behaves as if it were inside the VPC (AWS Bedrock documentation). The traffic is private. Bedrock is still an AWS-managed service, and the model is still AWS's to run. Google's VPC Service Controls likewise create a service perimeter around its managed AI services to limit data exfiltration (Google Cloud). That is a control around a managed service, not evidence that the model runs on your hardware.
Residency is a third, separate axis. Microsoft's Foundry documentation distinguishes where data rests from where inference is processed: Global deployments can process a prompt in any Azure region, Data Zone deployments hold processing inside a specified zone, and geography-based deployments keep it within the chosen Azure geography (Microsoft Learn).
So a private network path does not imply Swiss-only processing, and Swiss-only processing does not imply that you operate the model. A procurement team that asks only "where is the endpoint?" will get an answer that sounds reassuring and settles nothing. The better question is where prompts, retrieved documents, embeddings, logs, backups and inference are each processed, by which entities, under what retention and support-access arrangements.
What you gain, and what you take on
Self-hosting gives you direct control over how the model runs and how data moves through the system. Whether that improves confidentiality depends on the controls you put in place.
With the model server under your control you decide which weights are deployed and when versions change, which quantisation format is used, what gets logged, whether internet egress exists at all, which retrieval stores are reachable, how long prompts persist, and whether inference keeps working when the external network does not. Deeper customisation also gets easier: serving frameworks such as vLLM support several quantisation methods and distributed tensor- and pipeline-parallel inference, so a model can be split across GPUs when one accelerator has too little memory (vLLM documentation).
That control comes with ongoing work. Your team or an operating partner must own updates, capacity, monitoring and incident response.
The platform is bigger than the model
A production private AI system is not a GPU with a model on it. The path runs from the user through single sign-on and identity, into the application, through a gateway that applies policy, out to retrieval and tools, and only then to the model server and its GPUs. Around that sit vector stores, document storage, secrets management, encryption keys, audit logs, monitoring, evaluation pipelines and deployment automation.
If you run it on Kubernetes, the GPUs themselves add a layer. Kubernetes exposes accelerators through vendor device plugins that advertise resources such as nvidia.com/gpu (Kubernetes documentation), and NVIDIA's GPU Operator manages the drivers, device plugin, container toolkit, node labelling and GPU monitoring that make those nodes usable (NVIDIA). These are established tools, but someone still has to maintain them and respond when they fail.
"We can download the model" is not a production-readiness test.
Application permissions, retrieval and evaluation remain your responsibility in either setup. Self-hosting also adds model-serving and GPU operations.
Size GPUs from load, not from parameter count
Model size is a first approximation. It is not a capacity plan.
Google's published Gemma 3 figures show how much precision matters before concurrency even enters the picture. The 27-billion-parameter model needs roughly 54 GB for BF16 weights against 14.1 GB quantised to INT4; the 12B model falls from about 24 GB to 6.6 GB, and the 4B model from 8 GB to 2.6 GB (Google Developers). Those are weights only. Google Compute Engine currently lists NVIDIA H100 instances at 80 GB of GPU memory per accelerator and H200 at 141 GB (Google Cloud).
It does not follow that a model with 79 GB of weights fits on an 80 GB card. Runtime memory also carries the key-value (KV) cache used by attention, activations, framework overhead and batching, and long contexts with high concurrency push that well past the weight file. When a model will not fit on one accelerator, vLLM supports tensor parallelism across GPUs, and pipeline parallelism across nodes when it will not fit in one server (vLLM).
The procurement rule that follows is to size infrastructure from measured peak concurrency, prompt length, output length and latency targets. Quantisation lowers the memory bill, as Google's own figures show, but it has to be benchmarked against your quality requirements rather than treated as free compression.
Small models change the shape of the problem
Not every private workload needs an H100. Gemma 3 launched with 1B, 4B, 12B and 27B parameter models (Google Developers), and Google designed the 1B variant specifically for on-device deployment (Google Developers).
That makes edge deployment credible for narrow, bounded tasks: classification, field extraction, a local device interface, summarising something that should never leave the machine. The mistake is extrapolating from those to general business reasoning. Choose a small local model because your testing shows it is good enough for the defined task, not because private is assumed to be better.
The economics of hosted tokens
The economic case for private AI usually arrives pre-simplified: APIs are expensive, owning GPUs is cheaper. At sufficient scale that can be true. As a general claim it is not.
Start with what hosted inference actually costs. OpenAI's published standard, short-context list prices, checked on 13 September 2026, are 10 dollars per million input tokens and 50 per million output for GPT-6 Astra, 2 and 12 for GPT-5.6 Terra, and 0.20 and 1.20 for GPT-5.6 Luna, with a 10% uplift on regional data-residency endpoints for models released on or after 5 March 2026 (OpenAI). Anthropic lists 2 dollars per million input and 10 per million output for Claude Sonnet 5, and 1 and 5 for Haiku 4.5, while US-only inference for Claude 4.6 and later models carries a 10% premium; partner platforms set their own regional prices (Anthropic).
The examples below show how much API costs vary by model. They do not imply that a self-hosted model can match each model's quality. All amounts are in US dollars.
| Model, at 80% input / 20% output | Blended cost per 1M tokens | At 1B tokens per month | Spend matches an 8×H100 node at |
|---|---|---|---|
| GPT-6 Astra | $18.00 | $18,000 | 1.58–3.59B tokens/month |
| GPT-5.6 Terra | $4.00 | $4,000 | 7.09–16.15B |
| Claude Sonnet 5 | $3.60 | $3,600 | 7.88–17.94B |
| GPT-5.6 Luna | $0.40 | $400 | 70.93–161.49B |
Blended figures calculated from the vendor list prices linked above, before caching, batch pricing, tool costs, regional premiums or negotiated rates. The node reference is explained below.
In these examples, the choice of API model shifts the spend-equivalence point by a factor of 45. Name the model you would replace and test its alternative on the same tasks before drawing a cost conclusion.
What a GPU node actually costs
Public GPU pricing varies by region, reservation type and consumption model. Google's accelerator-optimised pricing table lists the eight-H100 a3-highgpu-8g configuration at roughly $88.49 per hour on demand and $38.86 with a three-year resource commitment (Google Cloud). At uninterrupted operation, taking 730 hours as a representative month, that is about $28,400 to $64,600 of compute per month before any application engineering, storage, networking or staffing. Google documents A3 High at its top configuration as eight H100s and 640 GB of GPU memory (Google Cloud); a smaller model may need far less than that. The lower price requires a three-year commitment. These are reference prices, not a quote for a Swiss region; check location, capacity and contract terms before budgeting.
With a pay-per-token API, inference spending falls when demand is quiet. A reserved cluster costs nearly the same at 10% utilisation as at 90%, and Google notes plainly that running instances accrue uptime charges while idle and that committed resources carry their commitment fees regardless (Google Cloud).
Self-hosting gets easier to justify as demand becomes both high and predictable. Neither alone is enough.
Spend equivalence is not break-even
The crossover column above equates API spend with raw node spend, which is deliberately not the same as a break-even point. A real comparison has to answer whether the open-weight model reaches the quality the task needs, whether that specific hardware sustains the required throughput and latency, what utilisation you will actually achieve, how many replicas high availability demands, and what engineering, security, storage, networking, backup, observability and support add on top.
The hosted side deserves the same rigour. Prompt caching, asynchronous batch pricing, enterprise discounts and routing cheaper tasks to cheaper models all move the number. Microsoft's Foundry Batch deployments advertise a 50% reduction against Global Standard in exchange for asynchronous processing with a turnaround target of up to 24 hours (Microsoft Learn), and Anthropic's Batch API applies the same 50% discount to input and output tokens (Anthropic). If your workload can wait for batch processing, include those discounted rates in the comparison.
The comparison that means something is cost per successful task at the required quality and service level, not cost per token.
In practice, self-hosting is hard to justify for sporadic internal assistants, pilots with uncertain adoption, work that depends on frontier capability that changes every few months, or an organisation that would be standing up a GPU platform team for a single application. It becomes plausible for stable, large-volume extraction, classification, summarisation or retrieval where a smaller open model has already passed your own evaluations and the hardware can stay busy.
Self-hosting does not make a system secure
An on-premises model server with a default password, broad network reach, unpatched container images and an unrestricted retrieval connector is less secure than a well-configured managed platform. Hosting location alone does not establish security.
Security is a property of the system, not of the building the GPUs are in.
AWS states the underlying principle for Bedrock: AWS protects the infrastructure, and the customer remains responsible for their content and for configuring the service, including identity and access controls (AWS). With self-hosting, your team also takes on security for the model servers and their infrastructure.
Generative AI also brings risks that a network perimeter does not address. NIST's Generative AI Profile extends the AI Risk Management Framework to this class of system (NIST), and OWASP treats prompt injection as an application-security problem in its own right (OWASP). For a private knowledge assistant, the review has to cover authorisation applied before retrieval so the assistant cannot surface a document the user could not open directly, prompt-injection handling for untrusted documents and web content, model and container supply chain, secrets and tool permissions, log retention and redaction, egress, key management, model provenance, and human approval on consequential actions.
Swiss and EU rules follow the use case, not the server
Hosting a model in Zürich does not demonstrate compliance. The Swiss Federal Data Protection and Information Commissioner states that the Federal Act on Data Protection applies directly to AI-supported processing of personal data (FDPIC), and the FADP itself sets out data-protection-by-design and by-default obligations and governs processors, security and cross-border disclosure (Fedlex). You still have to know what personal data enters prompts and retrieval indexes, why, who can reach it, what is logged, which processors are involved, and whether the controls are proportionate.
Nor is the FADP a blanket rule that Swiss personal data must physically stay in Switzerland. It regulates cross-border disclosure under defined safeguards rather than prohibiting it (Fedlex). A Swiss-only architecture is the right answer when legal analysis, a contractual commitment, sector regulation or a customer requirement makes it so, and it is not a substitute for doing that analysis.
GDPR also places conditions on international transfers. Chapter V sets conditions intended to keep the level of protection attached to personal data when it leaves the EEA, which the European Data Protection Board describes as protection travelling with the data (EDPB). A private model may reduce the number of external processors and international transfers involved. It does not remove processor governance, security-of-processing or assessment obligations.
For the EU AI Act, the 2026 calendar has moved. Regulation (EU) 2026/1744 confirms 2 August 2026 as the general application date but postpones Sections 1 to 3 of Chapter III, except Article 6(5), for the major high-risk categories to 2 December 2027 for Annex III systems under Article 6(2), and 2 August 2028 for Annex I systems under Article 6(1) (EUR-Lex). For the architecture decision, the key point is that classification follows what the system does and the role your organisation plays, not where the weights sit. A locally hosted model doing regulated high-risk work does not become low-risk because no vendor sees the prompts, and an ordinary internal assistant does not become high-risk because it calls a managed API.
Three architectures that work
A small company with a private knowledge assistant. Employees ask questions across policies, contracts and internal documentation. The path runs from the employee through single sign-on to the web application, through an authorisation and policy layer, to retrieval over an approved document store and vector index, then through a model gateway to a managed regional model. Add encryption, access-controlled retrieval, application audit logs, prompt minimisation and an evaluation set. The gateway is the important piece: it lets you introduce a private model later without rewriting the application. Evaluate managed regional processing or private connectivity before buying GPUs, because they often satisfy the actual confidentiality requirement at a fraction of the operational cost.
A medium enterprise running several AI workloads. Users and applications reach an enterprise AI gateway, which applies policy and data classification and then routes. Low-sensitivity work and anything needing frontier capability goes to managed models. Sensitive, high-volume, predictable work goes to a private Kubernetes inference plane running vLLM over a GPU pool. Both paths share identity, secrets, observability, evaluation and cost telemetry. The commercial advantage is that no single model or vendor decision gets locked in prematurely.
A regulated organisation. Managed employee devices and enterprise identity feed a controlled application zone, then a policy and data-loss-prevention gateway, then authorised retrieval, then a private inference cluster with no default internet egress, allowlisted model-artefact import, a private container registry, HSM or KMS-backed keys, an immutable audit trail, a restricted admin plane, a controlled update pipeline and a separate disaster-recovery environment. High-impact actions stay behind deterministic validation or human approval rather than autonomous execution. The architecture improves control over data flows; it does not replace governance documentation, risk assessments, supplier controls, model evaluation, security testing, incident management or AI Act role analysis.
Managed inference
The sensible default. Fastest to deploy, elastic under uneven load, and where the strongest models are.
- Idle capacity stays the provider's problem
- Caching and batch pricing cut the bill further
- Regional endpoints answer many residency requirements
Hybrid behind a gateway
One policy and observability layer, two inference paths, routed by sensitivity and cost.
- Sensitive and high-volume work runs privately
- Frontier capability stays available for the rest
- The model decision remains reversible
Fully self-hosted
Justified by a control boundary, a latency requirement or sustained utilisation, not by preference.
- An open-weight model has passed your evaluations
- Demand is high and predictable enough to keep GPUs busy
- You already run Kubernetes, GPUs and on-call competently
Should you self-host?
Use these questions to assess your workload and operating capacity.
| Question | What a clear yes points to |
|---|---|
| Must prompts and retrieved data stay inside a boundary you control? | Private or self-hosted |
| Must inference keep working while disconnected from external networks? | On-premises or edge |
| Is latency at a site, factory or device the binding constraint? | Edge or private |
| Has an open-weight model demonstrably met the quality bar on your data? | Self-hosting becomes possible |
| Is utilisation high and predictable enough to keep GPUs busy? | Self-hosting economics improve |
| Do you need model-level customisation no managed service offers? | Private or self-hosted |
| Do you already operate Kubernetes, GPUs and MLOps competently? | Self-hosting risk drops |
| Is traffic low, bursty or genuinely uncertain? | Managed API |
| Do you need the newest frontier capability now? | Managed API |
| Would this create a 24×7 infrastructure discipline you do not have? | Managed or hybrid |
| Are only some requests sensitive? | Hybrid, with a router |
| Is the only stated reason "self-hosting is more secure"? | Reopen the requirement |
Self-host when control, residency, offline operation, latency or sustained utilisation produces value that exceeds the capability gap and the operating burden. Otherwise buy managed inference, or run both behind one gateway.
Where AI is the wrong mechanism
Self-hosting cannot rescue a poor use case. A language model is usually the wrong core mechanism when a deterministic rules engine solves the problem more reliably, when every answer has to be exact and nothing independently validates it, when there is not enough authoritative data to support the decision, or when an autonomous action can cause disproportionate financial, safety or legal harm without an effective control in front of it. In those cases AI can still help with retrieval, drafting or triage while the consequential step stays deterministic or human-owned. That boundary is the subject of our guide to identifying AI use cases that are worth building.
Where to start
Begin with the workload, not the hardware. Write down what it does, how much of it there is, how predictable the volume is and what latency it needs. Classify the data that enters prompts and retrieval, and establish the legal boundary that actually applies rather than the one assumed in the meeting. Test whether an available open-weight model reaches the quality bar on your own examples, then compare a managed deployment, a private one and a hybrid against the same requirements with utilisation and engineering cost included.
Build the smallest architecture that satisfies the constraints, and put a gateway in front of it so the model decision stays reversible.
At Alpine Edge, a technical assessment brings together the workload profile, data classification, requirements confirmed with your legal team, quality benchmarks and a costed recommendation. We then build and operate the chosen solution through our AI integration and private AI services, supported by cloud infrastructure and DevOps.
Sources and pricing assumptions
- OpenAI API pricing. Standard short-context list prices and the regional-processing uplift, read 13 September 2026.
- Anthropic Claude pricing. Model list prices, the Batch API discount and regional endpoint premiums, read 13 September 2026.
- Google Cloud accelerator-optimised pricing. On-demand and three-year resource-commitment rates for a3-highgpu-8g, checked 13 September 2026.
- Google Cloud Compute Engine pricing. Uptime charges on idle instances and committed-use fees.
- Google Cloud GPU machine types. H100 and H200 memory per accelerator, and A3 High configuration.
- Gemma 3 quantisation figures. BF16 against INT4 weight memory at 27B, 12B and 4B.
- Introducing Gemma 3 and Gemma 3 on mobile and web. Model sizes and the on-device variant.
- vLLM documentation and parallelism and scaling. Quantisation support, tensor and pipeline parallelism.
- Kubernetes GPU scheduling and the NVIDIA GPU Operator. Device plugins and GPU node lifecycle.
- Microsoft Foundry deployment types. Global, Data Zone and geography deployments, and Batch pricing.
- Amazon Bedrock VPC endpoints and data protection. PrivateLink access and shared responsibility.
- Google Cloud VPC Service Controls for enterprise AI. Service perimeters around managed AI.
- Swiss FDPIC on AI and data protection. FADP applicability to AI-supported processing.
- Federal Act on Data Protection. Design obligations, processors and cross-border disclosure.
- EDPB on international data transfers. GDPR Chapter V transfer conditions.
- Regulation (EU) 2026/1744. Revised EU AI Act application dates for high-risk systems.
- NIST AI Risk Management Framework. Generative AI Profile.
- OWASP prompt-injection guidance. Application-layer AI risk.